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'ABSTRACT » 

Since the usefulness of the Inter service Procedures 
for Instructional Systems Development (IPISD) depends on authoring 
aids, which enable persbnnel to translate IPISD procedures into 
instructional pro^cts, this project examined the fe?tsii!?ility of 
providing Auch "hq?^o do it" . guidance for the instructional design 
and de^elopaent tasks idejitified for Block II. 2 -Develop Tests and 
Block III. M-D^velop Instruction. Online authoring aids were developed 
to be used on the PLATO IV cinputer-Assisted Instruction (CAI) 
systea, as well as off-line versions, to assist the a'Uthor in 
preparing materials for both CAI and non-CAI delivery of instruct ioy: 
The three X4fel^ of evaluation conducted included an informal 
evaluation Mi existing IPISD materials, a formative evaluation of the 
newly developed" authoring aids, and an evaluatioi^ of the 
instructional materials produced. Findings were positive--user- — ^ 
acceptance of the authoring aids was hi^h, and the tine required for 
development.*of test and lesson material was significantly reduced. 
Based on these findings, *he development of such aids for additional 
blocks of the IPISD model have been initiated. Flowcharts ' for both 
the two blocks studied and the aut,horing aids are provided, and a 
97- item bibliography is attached.. (BB H) 
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FOREWORD 



The Computer-Based Education^^^echnology Team of the U,S, Army 
Research Jnstitute for the Behavio^^Wand Social Sciences (Afti) performs 
^ research and development in eares of educational technology that apply to 

military training • Of interest are computer-based instructional delivery 
* systems that focus 6h developing the accompanying instructional course- 

ware in th^: m6st efficiAit and cost-effective manner possible. Develop-- 
^ ment and implementation of suijfh systems will help solve 'the problem of 

training individuals to prodXice good courseware in a reasonable time, at 
an acceptable cost. ^ 

^ . / ^ ^ 

TJni^ Technical Report describes a development And feasibility demon- - 
stration of tw9 author aids designed to assist individuals in developing 
test* and instruction. The project was funded jointly by ARI and the 
Defense- Advanced Research Prdjects Agency (DARPA) • To accomplish this 
research, ARI • s resources were augmented by contract DAHC 19-76-C-0041 with 
the Human Resources Research Organization, an organization selected as hav- 
ing unique capabilities for research and development in this area. 

Personnel at the U.S. Army Engineer School (USAES) , Fort Belvoir, Va., 
provided guidance and assistance throughout the project: Dr, Everett 
Rompf, Mr. Jack Ainswor€h, LTC Ernest Larson, MAJ John Harvey, MAJ Ramile R. 
Rebello, ILT D. Bunn, SFC, Alton J* Blanchard, and SFt Leon Loomis. In 
addition," Dr. James Kra^tz, PLATO Services Organization, Computer-Based 
Educational Research Laboratory, University of Illinoi/s at Urbana, and 
^s. Beverly Hunter and Mr. Richard Rosenblatt HumRRo' also contributed 
to the research effort. 

The entire resear6h work unit area is responsive to the requirements' 
of. TDT&E Project 2Q762717A764 , "Educational and Training T^echrtology , " the • 
1977 ARI Work Program.' v / 

EPH ZEiWIr 
hnical Diifector , 
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ON-LINtt AiItHORING AIDS FOR INSTRUCTIONAL DESIGN 

V • r 

BRIEF ' _ 



Requirements ♦ ' 

The purpose w^ats to examine the feasibility of providing "how to dp 
it" guidance (authoring aids) f?!>r the instr\;^ctional design and development 
tasks identified by the Interservice Procedures for Instructional Systems 
Development (IPISD) model. The .usefulne$s of the IPISD model depends on 
authoring aids Which enable training personnel to translate IPISD pro- 
cedures into instructional products. The authoring aids developed by this 
research should be useful for computers-based and off-lin«^ instruction and 
be general izable to differing subject matter areas. 

Procedures * I * 

^Authoring aids were constructed, implemented, an* tested. The author 
ing^aids were developed ^o be used oh the PLATO IV Comi^teiP^Assisted In- 
struction (CAI) system. - The first step prodi^d flowcharts which detailed 
the steps of the IPISD Blocks II. 2 (Develop Tes1:s) and 111,4 (Develop in- 
struction). I On-line authQ> aids aa well as off-lirte versions were produped 
to assist the author in preparing materials for CAI and hon-CAI delivery 
of instruction. . , ^ 

Three levels of evaluation were conducted. An informal evaluation 
on existing IPISD materials was performed, ^and a formative evaluation on 
the newly developed authoti^g aids. Finally, the instructional ^materials 
were evaluated by military authcjrs and administgred to O.S. Army Engineer 
;5chool trainees. 

Findings: ' ^ 

The feasibility of on-line aids. for implementing IPISQ-Blocks II. 2 
(Develop Tests) and III. 4 (Develop Instruction) was demonstrated through 
thri evaluations. User acceptance of the aids was hifh, and the time re- 
quired for development 6£ test ancj lesson material has been significantly 
reduced. v ' > ' • - . 



Utilization of Findings: 
♦ 

Based on these findings, * the development' of authoring^ aids for addi- 
tional blocks of the IPISD model was initiated. 
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> The pi^rpo9e- of this -chapter is to provide j^n overview of the «tate~ 
of-the-art related- to Instractlonal systems dc^s^gn and evaluation. ^lajor 
subjects addressed In this chapter are: ( 

' • ' \ * 

• Systems Approach to Training . . \ 

• Problems of Implementing Instructional System Development Models 

• The Need foi: Author Aids ^ 

• • Approaches and Techniques tor Evaluating Instructylon 



SYSTEMS APPROACH TO TRAINING. 



A revolution In the technology of training within the military and 
industry began when the^ sy^tem^ concept was applied to the dei^elopment and' 
conduct of training, , Even now, after 25 years, the full potential of 
app^lng the systems approach to improve the effectiveness of instruction. 
Improve on-the-job performance .and lower the. cost of training has not been 
realized. Even so, results from applications of the systems approach in 
terms ot. improved Instruction, incre^s^d relevance J^n what is taught, and 
lowered costsr have b0en so dramatic that, at present', the systems approach 
to training has perme^ited civilian training (and education as well) — 
train^tng in business and Industry j . training in the military services, and 
training in other agencies of the, federal government. • 

Many different names; terms, and variations are or have been used for 
the systems approach to training. Some, of the- names arie: "systems 
engineering of training," "curriculum engineering," "systems approach to 
tr^itHlxlg," "instructional systems development,^" "training situation analysis," 
"modern Instructional tecHnology." Even among those using a particular 
name, there are many divergences in de^ir^ltions , particulars of technique 
and procedui^es , jand effectiveness with which the systems approach is 
applied. Despite such variations, the^dormon thrust and orientation of 
these appllfcatlons is pre-eminent, especially as they contrast with tradl-- 
tlohal approaches to training. / 

'•'••/' 

The essence of the systems apptoafch *to draining rests in identifying 
expUxrCt . end stated that are to be achieved through training and in defining 
sets of orderly, objective, and explicit procedures; to do that which is 
necessary to achieve these end states' in. the most cpmprehenslve, reliable, 
effective ^pd efficient manner. 



The systems approach defines a p 
that is ultimately to be per^formed 



and 



roCess \fj\\ic\\^ focuses upon the job 
upon the individual who is to learn 
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to perforrt that job. Traditional . approaches, by contrast, focus upon 
conventionat aubjeat matter hloake that tend to be more of the "school 
catalog" variety and are, generally, only approximately pertinent to what, 
the student will be doing later. 

^ In addition, most traditional approaches place the burden for Infor- 
mation transfer upon the studerits rather than on the instructional 
materials. Wliether the ^struct ion is rapid or slow, complex or-dulL, 
the sttident must adapt . contrast, in the systems approach, it is 

feasible to engineer flexibility into the instruction and, so, to adapt 
the Instruqtional system to individual differences among the" students . 
Special consideration is given to: v ^ 

» • Evaluation of the needs of each individual student, 
• ^The iiature of instructional content to be imparted, \ 
m The instructional decision rules that mediate between student 
nee^s and instructional content. 

The systems approach is just what. the name implies: a systematic 
process for specifying the desired products of graining apd selecting what 
will be taught, how it will be taught, what the presentation mechanism will 
be* and evaluating the effects of each phase of the process. It focuses - 
on student^ performance as a determinant of content. Its proper application 
can hardly fail td improve instruction where only incidental attention has, 
been given to these functions. Thus, in the systems approach setting, 
.uncoi^ventional clusters of instructional material ipay be used for a uniform 
(usually small) group of students, each of whom is being prepared to perform 
the same job. Major . efficiency is achieved by directing instruction ^ 
precisely to the student and to what the student will use on the job, 
thereby assuring relevance and efficiency, precludiAg oversights, and 
adapting instruction to the individual. 

During the past 2C\ years, many attempts have been made to codify s, 
.definitive technology f6r the systems approach to ^training. Early efforts • 
in this area included those by HumRRO on behalf of the U.S. Army in the 
• early 1950s and the development of the USAF personnel subsystem approach in the 
mid-1950s. tThe HumRRO mode], for example [1,2*3,4], is a seven-step pt^ocess which 
starts with\the development of a man-^nvachine system analysis model. From 
that, a job model is developed w;hich then leads to both the specification 
of knowledge and skillie required for adequate performance of the job, and 
the proficiency test development. The proficiency test measures the \ 
ability of the student to perform actual job tasks, thereby assfessing the 
job proficiency of the student. From the specification of skills and 
• knowledge, one* may determine the instructional objectives, which is tfo say, 
those specific requirements for an instructional program. Once the 
instructional objectives have been determined, then ^ training program 
can be constructed. The seventh step is the evaluation of the training 
program. • 

One o"f the more notable of recent isy stems approach efforts is the 7 
InterService Procedures for Instructional Systems Development (IPI9D) Model 
[5,6,7,8,9].' This \nodel was prepared by the Center for Educational- 

O . ■ ^ 2 ' 8 ^ ' , 
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Technology at Florida Stk^te l/niverslty under contract with the Inter- 
^ service Committee for Instructional Systems and Development, involving 
* the Array, Navy, Air Force dnd the Marine Corps, The IPISD contains/^ 
standar<fl2ed rationale, fermltiology, and basic concepts of ^instructional 
systems. These have evolved by developing and receding the result.s of 
efforts in theoretical and guidance materials required for actually per- 
forming instructional systems development. Prior to this effort, the Air 
Force had undertaken a large activity to develop, define', and record a 
deiinitive technology for instructional fl^stems development [10], and the 
^Army had embarked on an ambitious five-year program to systems engineer 
all of Its training courses [11] . Some of the development of. the systems 
approacli to training has gone on outside the Services, particularly in 
industry [12,ia,14]'. In addition, Mager [15] and others such as Glaser 
4l6], Amm^rman [17], Krathwohl [18], Bloom [19], Melching [20], Gagne [21], 
Esbensen [22], Bond [23], and Butts [24], to mention only a few, have 
made significant contributions to systems approach modei^s through their . 
research in the development of behavlotal objectives and sequencing of 
instruction. In the Navy, much of ^-the work dealing with the. systems 
approach has been carxied out by USNTEC with reference to simulation 
('e.g., [25,26]). ^Iti ajdition, the.Navy has initiated several major 
effoi;ts related to training systems design of a more general nature [27, 
28, ?9]. 

The IPISD model shows promise as a tiseful tool in instructional 
system development activities and is presently undergoing preliminary field 
evaluation. The model consists of five major phases which can b^ C9nceived 
under the -ADDIC rubric: 



A 


analyze 


D 


design 


D 


Jevelo^ 






-2 


implement 


C 


control 



Figure 1 ^s a breakdowil of the five phases into more detailed activities 
(blocks) comprising each phase. 

PROBLEMS or IMPLEMENTING INSTRUCTIONAL SYSTEM DEVELOPMENT MODELS ' 

Early applications of the syatems approach to training were accQjnplished 
by 'expert training developers. In the 1960s, the possibility of having 
laymen use these models to achieve the success of the experts, by imitating 
their actions, was fexplored. The use of an ISD manual by existing military 
pei;ponnel v^ith little or no 'experience in training program design may ^ost 
a fraction of the Qo^t of hiring or fontractirig experts to do the deve^praent 
Even so, the cost/effectiveness of the model will still depend on the \ 
effectiveness of the .model, or tools, in enabling laymen to produce 
effective instruction. 

I " 3„ 
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THE BLOCKS IN EACH PHASE ARE: 



I 1 

►ANAlY/t 
JOB 



1 ? 

SFLFCT 
' 1 ASKS 
♦ UNCTIONS 
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CONST lUJCT 
MOB 
PERFORMANCE 
WeASUBES 



I 4 

ANALYZE ^ 
^EXISTING 
COURSES 



1.5 

SELECT 

Snstructional 

SETTING 



> DEVt I OP 
OBjtcilVtS 



n 2 

► DtVEl.OP 
TtSlS 



n J 

OESCRIBE 

CNTRY 

BEHAVIOR 



n 4 

DETERMINE 
SEQUENCE & 
STRUCTURE 



in 1 

SPECIFY 
► LEARNING 
EVFNTS/ . 
ACTIVITIES 



111 ? 
SPtCll Y 
INSTRUCTION 
MANAtUMENT 
PLAN 8* ' 
on IVFRY 
SYSl I M 



m 3 

REVIEW/SELECT 
EXISTING • 
MATERIALS 



mi 

01 VKOP 
INSIRUCTION 



► VALIDATE 
INSTRUCTION 



J 



LV 1 

IMPI IME^NT 
INSII^UCTIONAL 
MANAGE Mf NT 
PLAN 



IV 

i;ONl)UCI 
INSI f^UOTlON 



^ CONDUt T 
' INTL RNAL 
f VALUATION 



^coni^uc:t 

EXl! RNAL 
\ VAL UATION 



2 3 

► REVISE, 
SYSTEM 



A Detailed Breakdmvn of Activities to be Performed in EactyPhase 

' ^ (TRADOC'PAIf 350-^30, Executive Summary and Model, August 1975) 



Figure 1 



In the past few years, ^problems with attempts at implementins ISb 
models by laymeu have aurfaced, Montmerlo [30] conducted^ a comprehensive 
review of ISD state-ot-aa;t and problems of implementation. His conclusion 
was that "available ISD-type methodologies will 'not' allow the layman to be 
as* successful as the expert (The particular IPISD manuals with which we 
are concerned, however, were not > a 'part of this review,) 

Montmerlo cites the paper by Ricketson, Schulz and Wright [31] as ihe 
"most significant article concerning the problems of fSD," because 
represents "the only empirical evaluation of^an ISD-type methodology," 
Ricketson, Schulz, and Wright studied the CONARC REG 350-XOO-r an4 its 
implementatichi by Army instructional developers* Although IPISD is intended 
to be a considerable improvement over 350-100-1, many of the same problems* 
do apply. For example, Ricketson, et al > , found that "High rates of 
personnel turnover 'within some curriculum development groups have resulted 
in a general reduction of systems ^engineering program productivity," 
Assuming this to be a continuing Reality in military instruct ional^<^velopr 
ment, the need becomes evident to provide*authoring aldjg which can be 
quickly learned by new <|evelopers. The study found, among other things , . 
that developers tended to develop training programs that employed the 
same techniques^ with which they had been taught, gince they did not have 
the ability to assess other training techniques and equipment, 

/ / • 

IPISD ffnd other ISD manuals are intended to hftve general applicability. 
However, it has been clearly recognized^ by many experts that the sayie 
^Tethodologies cannot be applied to the univ4rse of training problems. The 
literature on task analysis, for example, contains a number of articles on 
the impossibility of using the same method for all tasks [32,33,34], 
The IPISD Executi\je Summary & Model [9] .also emphasizes the need for 
diffei*ent methodologies in the statement, "The extent that on/e used the 
interview method, the observation method, or the occupational survey method 
depends on the nature of the job being analyzed, the jpb data already 
available, and the availability of analyses -resources, " [9] ^ 

t 

While the "whatk to do" may remain relatively constant across training 
problems, the "how to do it" may vary enormously. This again is why the 
instructional systems designer needs a wealth of aids to refer to in 
dealing with a specific training problem in a specific subject area,^ 
While the IPISD manuals do provide far more References to the literature 
than previous manuals did, they do not provide specific "how to do It" 
guidance for specific design and developm%|^ tasks, [9^ p. 124] 
♦ 

Another major problem area of ISD ^implementation in general is the 
management of the instructional development process. For example, when a 
Change is made in the conditions of a part icular test item, this has many 
ramifications backward and forward in the ISD i^ro^Jess, The management of 
these changes, including the communication among various members of the 
IPISD teqm, is complex and usually ^requires some management aids. 
Discussions with training staff at Ft, Belvolr provided us with practical 
evidence that it is in the area of management of the ISD process that 
major, problems continue to be found, • 



Other problems of using IPISI^ relate to the background skills of the 
tea!n|Lfeader and of thle members of the development team. This is pointJed 
out in the IPISD Executive Summary [9], In the Mon^merlo study [30], and 
in many other sources [35,36,37], • " 

The IPISD model is advanced over other systems ,apptoacl| models in 
providing guidance to the training ipanagex* However the I^ISD manuals 
are not presently intended to provide specific procedures for every instruc- 
tional situation that can be encountered. Some situations are now covered' 
only by the general principles underlying the Model. If IPISD is to have 
a "fair ch«nce" of beirtg accepted by training managers, it is essential 
that tools and authbr aids be developed that will permit training personnel 
to readily and effectively translate recommended IPJSD procediires into 
meaningful instructional products. This rationale forms the basis for 
the initiation of the present project. 

THE NEED FOR AUTHOR AIdS ^ * 

Author aids are any products used in accomplishing 'one or mOre. steps 
of the IPISD procedures. Under this definition, thousands of guidebooks, 
research studies, texts, professional articles,^ rand technical, repojrts 
could be considered- as aids. If an instruct lx)t)ar*5?^$htem deSlgn^'V wete 
familiar with the full range of aids^ av.ail^b3:i|V, h<» -w»xil'd In 'fact, be an . . 
expert in the fiejd of IPISD and theVe^o^e V^rVot b'e V subj 
The probl^ for the author (any member of the rfev^lppift^t iS"'t6 / 

know what^ids to use when, to know they exist / to/T^SVte acc^i^iB to them in 
a timely way, and to have some facility and jiWgm'ent ip tfi^ir application. 

In thi3 effort two of the blocks of the IPISD model vete selected- - 
for evaluation. Block II. 2, Develop Tests, and Block III. 4, Develop 
Instruction. The first block, II. 2, was required in the RFQ. Our^ 
choice of Block III. 4 is ^P^eed on a number of mutuallyisupportlve 
gtineral and specffic re'asons. Many leading instructional technologists 
and designers have concluded from their experience thai^ the Development 
Phase of instructional preparation is the significant component of the^ 
systematic approach to producing quality instruction. It 1^ expensive, 
time consuming, \cr it leal, and require^ specialized capabillt iea^. Van Pelt 
and Rich [38], flor example, speak from experience in the Army trainings 
environment: "There is no question that mucVi time is wasted by y/riters 
casting about fotr a reasonable set of guidelines to follow that will result 
in lessons requiring a minimum of editing and r^visi-on." For the Navy, 
Aagard and Braby [39] have emphasized the need for an algorithmic approach 
to translate basic learning events into* instruct ionally meaningful task j 
categories ". . / in a manner that emphasizes the flow of events and the 
combining and sequencing of learning guidelines in the design of a training 
program. . ." (p. 7). In the civilian sector, Lipson [40} has stressed the 
need for ". . . increased investment in developih^t pf instructional . 
materials.'* The "homemade** variety doesn't have . . the qi^aliti^s of^ 
craftsmanship, artistry, nor the proper incorporation of what Is known 
about effective instructional design to be widely used." Industrial 
developers of CAI (Slmonsen and Renshaw [41]) have stressed that • • 



the cost of lesson pr^paral; ion actually iftore thUn doulyles the projected . 
cost of -an hour of CAT and cannot be ignored*"' Recent analysea (formal' ; 
ahd informaiy by the Training 'bnd^ Evaluation Group (TAEG) If support this 
assertion fbr the general cijis^ of individualized Inkt rue tibn that ^wvolves 
a systematic approach to tfie development of materialsi^* . # • 

On a specific and practical basis we have learned ft^m th^e Curriculum 
Development personnel at tfle USAES that they^Viave experienced the most 
difficulty in using IPI^D with Block Ill\4, Dfevelqp Instrti^on. A 
comparison of. test constjruction • to vjgetieral d^v.elopraeijJfe of Instruction 
suggests that in develop^lng^ tests (IPISD Block II .'2), authors need help ' 
In perforining such activities as t^e follqwliJ^: • 

. # r Developing test items? that actually test the Terminal Learning 
/ Objectives (TLO) , Learning Objectives (LO) , And Lesson Steps ,<LS)* 

• Constructing h^nd^-on scorablei units of Skill Qualif^-cat ion ' 
Tests (SQTs) . , . ' • 



A 



f Developing scoring procedures. 

• Writing test item^ that will help identify bad instruction. 

• Devising test items that will support remediation strategies. 
m Generating test items &nd alternative forms of items. 

• Managing the test development process, e.g., have all TLGte been 
tested? . ' . 



• . Obtaining reliability measures on test items. 

-* * \* 

, • Determining tlie validity of test items. 

In B16ck III. 4, Develop Instruction, a variety of aids are needed. 
Soma of the activities and decisions "which require support include: 

• , Ensuring reading level is appropriate to the audience. ' ^ 

• Deciding what kind of drill and practice is neede^d, and how much, 
for a given task. 

• . ..'■,»' 

• Decid-ing how the student will be able to obtain additional help. 



Determining the nature, frequency, and type of feedback to provide 
to the student during the • instruction. 
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Personnel Communijcatlon, Dr. Richard Braby of the Training and Eyaluation 
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Aa an ov^riSlding concern, aids are needed to help reduce the time it takes 
to devel6p\uall,ty instruction attd teats, and to maj^e the development! 
process as efficient as posalbleN. ' ♦ . . 

A wide variety of aids exist. A Recently Wgan [42]. completed .a survey 
of exiating toola/procedures which <^|puld be,uaed by *natructional''developera 
in conjunction with the IPI^D. The Areaulta of this survey indicated that ^ 
aida exist for a number of IPISD componenta. Unf orti^atply , a considerabld. 
technical background, and level of expertise ia required for their uie. ) 
Thus; even when aids are availabOje^ th^rei remains the problem of uaing 
th^m without imposing an undue burden on the author. - ' » 

The existence of theie aids testifies to the recognition that they'are 
needed.' However, in the case of autKffr aids it ia,nQt the variety a'nd 
quantity in the. universe that counts. What majy:iarg l|s that the appropriate 
aid be available, easy to uae- and accessible at the rjight time and place." 

The majority of existing materials are more of the "what to do" nature 
than the "how to do it" variety. There are a number of general resource 
guidelines already available to aid systematic develop^nt^^^-Lrfstruction., 
The following discussion of these resources is illustrat*^ and not intended 
to be exhaustive of the field. Because they are genetal, many of these 
handbooks, manuals, etc.,. are difficult to categorize in terms of specific 
single phases of . IPISD. However, many examples qan- be categorized as giving 
guidance primarily in Analysis, Design, Development, Implementation, or 
.Contrdl. ' Fpr instance, Harless [43] emphasizes the importance of "front- 
end analysis" to solve human performance problems. For his target audience, 
primarily llnagement personnel, he reconmiends defining the problem, inves- - 
tigating its characteristics and studying alternative solutions aiid thejjf 
costs before making any decisions'. McKnight's Work [44] on tailoring 
military ilRraining by systems and job analysis i^vides another useful 
resource 'for the' Analysisj, .Phase . The recently produced Marine Corps 
training guide for task analysis, [45] is another such aid. ^■ 
/_.■•■. 

Mager's technique^ 446] are classic as aids to stiuplating precise 
instructional ob J ec'tiveSa^d thus fit in to. the Design phased of IPISD. 
The T AEG approach, to cdtfegoxizing instructional tasks according to 
particular l«arn&»g al^oritVims [§9] can in some instances be useful for 
/design purposes> \ / ; . ' * , 

brief b6ok by tip$' l^7] for beginners falls between Design and - 
ment . It descrlbesjttie systematic procedilir^s necessary* to begin 
'wrTtlhg. pragr|ims. -tie neglects the details such as frame writing or 
program format,- conEeht rating on the "practical" issues of steps that ^ , . 
precede writhing, testing programs, and avoiding pitfalls. Drumh^ller's 
ha^dbcrok [AS] Is eve^ broader. In his guide of curriculum design for 
individualizedfyihstru'ction hf has highlighted the need for materials to 
' "have built;- 15 compjfehensiveness." His systems approach provides guide- 
lines in the iorjn/^'f a detailed model for curriculum design. It includes 
defining. objectives, analyzing snb-objectives and integrating them into 
the learnlnK experiences. .Wong and Raulerson*s [49] guide spans all the 
' s'tQPS of IPlSD'^n brief but they spend more time and give more detailed 
•" ^ • ■/ • . 



aid for Deslgrv-and Developtaent. In the latter, they provide guidance for 
• media selefttlon based on .variations In stimulus requirements of the learning- 
,ta«ks. Their selection guide is. useful; and while not 'apparently theo- 
retically derivedv.from learning principles as is Brlgge' approach [50],- it' 
may be more helpful to the layman. As with most such selection guides,. 
• the user still must choose from two or three potentially equivalent alter- 
natives, Wong and Raulerson's model [49] is based in general on ia strict 
sequepti^l view of learning. 

A number of published texts ari available as aldsfor various parts 
of the Develo-pment Phase of IPISD. -'MarKle [51] has discussefd t^he cons true- 
^ timi, format, and sequencing of the frames in her texts, both of ^hich are 
pro^ammed.Sperry [52] with its Instructi^nral Program Del^elopment Wftrkbook . 
4ias developed and used comprehensive .plans' for a workshop (along with a ~ 
workbook) on instructional program development. Its structure follbws the' 
.philosophy of mat*hetics that includes demonstration, . prompt, and release 
exercises. Espech and Williams [53] in their Handbook for authors of 
programmed instruction describe the process of constructing programs with - 
major emphasis on editing, testing, and analysis. The finished pSduct is 
then assured of being a "packaged change of behavior." Hawkridge, Campeau, 
and Trickett [54] provide a rather unique resource to help the evaluator 
prepare his reports. While it is written towards a school systeih audience, 
its clear, concise approach should make it usable in a military context. 
More recently, and still under development, Hillelsohn is employing a 
programmed instruction approach to creating and managing computer-based 
learning materials [55] . This effort is expected to provide an additional 
means for implementing several components of the IPISD model. 

•The most relevant example for the Army as an aid to Implementation of 
instruction is the military training, manual , FM 21-6 [56]. Its format is 
readable and comprehensive, replete with examples for the instructors. 

Not a great deal of useful materials is available for the Control 
phase per se. However, Cogan's case study approach [57] is illustrative 
of such aids which could be a useful resource to training managers. 

Havii]g noted some of the available resources, certaiti aaveate are' in 
order. Rather than providing actual help in performing the authoring work, 
or even detailed "how to do it" guidance, most of the existing author 
ai^s may serve siijiply to reinforce or i)roaden the guidance provided in 
IPISD manuals. Aagard and Braby [39] very carefully note the practical 
limitations of the use for their -algorithms and guidelines to general 
approaches. "The task categories and related guidelines, are not at a ' 
level that will accommodate any training setting."' Briggs' handbook [50] 
.may also serve such a broad guidance function. Care should be exercised 
in the selection of aids to be integrated with IPISD, so that 
IPISD^ authors are not confronted with a confusion of different yet 
similar models, sets jargon, procedures or forms. Existing general 
manuals differ ^ from one ''another in that they: 
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• include different steps or different names for steps. ' 

• include -different methods of accompllshlpg each step. 

* » * * . 

• Provide different levels of dpecl^lclt/ In the detail Included^ 
X under each step*. * ' ' 

m Provlcte different format^ for reporting the work accomplished 
under each step.. ^ 

* • ' Th«s. tb. provide' the most- efficient alds.to IPISD Jev«16pers7 the guidance 
found In. somp of these manuals ijnd guidebooks needs t:o. be translated and 
■ Integrated Into, the IPISD framework, rather than referred to In ltd soured 
fonrt. " ■ ' ^ J 

•One example of a useful "how to do It" gylde- is the Guidebook for 
Developing Criter ion-Referenced Tests [5B 1 ^ To make the guidance In this 
book readily acc^lible and useful to tlve IPISD aUthor, the Ideas need to 
be Integrated lnt6. the IPISD framework. Another Important aid In the ^ 
' testing area is th^; recent Manual for Developing Skill Qu alification [59] 
and the Procedures for Validating Skill Qualification Tests [60]. 

•In some cases, the author needs actual assistance in performing an 
activity, rather than simply how to do it information. For example, 
automated readabilJLty ind^ices can take some of the, workload from. the author. 

Some aids are specific tc a particular method or theory of instruction. 
A TJi»e -layman author needs some basis for using that particular approach or 
method, and needs to know that it is one of several alternatives. Thus, 
for example, Matkle's [51] texts might be relevant and useful to an author 
who has decided to follow her particular approach to programmeft Instruction. 
Alternatively, Sperry's comprehensive plans for a workshop and workbopk 
• [52] on instructional program development as noted earlier follows .the 
philosophy of mathetics. n 

In the more specialized areas of computer'-based Instructional develop- 
ment, an array of automated aids l?ave been produced. The TICCIT prbject 
[61], for example, uses highly structured forms- for text preparation, apd 
highly proceduralized production techniques for aut4ioring teams.- Similarly, 
Prohect IMPACT [62-68] developed standard formatting aids for authdrs, 
sQphistieated techniques for logA and text Reparation, instructional 
management, etc. Again, these; aids are highly specific to a particular 
InBtructional strategy or method and (particularly in the case of TICCIT) 
frequently constrained by system hardware and software constraints. 

The array of automated aids for authors of computer-based materials 
Includes: special programming languages, test item generators, scoring 
algorithms, recordkeeping facilities, objectives data banks, text editors, 
graphics aids, student response (analysis algorithms, data analysis routines, 
statistical subroutine packages. 

O 10 ' t iiv 
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Among tfie major problems for users of computer-administered instruction 
(CAT) is the high cost of developing qurfllty instructional material.- The 
problem is exacerbated by the fact that new CM programmers -frequently 
te^ulre several months of, training before they, ana able to produce quality 
Instructlotj within an .acceptable time frame, . fhese individuals view their 
role aaji^nt of subject-matter experts and 'educators gather than programmers. 
To satiAfy this audience, authoring aitJs are, needed that will permit 
educatots U) rapidly develop quality Instruction without extensive CAl 
langiJage. training. " ' ♦ • . ' . 

As a first: step in meeting ^ht^ need, ^ind a forerunner of the prfesent 
research. HumRRO completed the develot>ment of s^ts of author aids called 
MONIFORMS [67,68]. These aids asdisfan author ia generating question-and- 
answer -type practice Items. The author interacts with the computer which 
leads the author step/-by-9tep in the creation' of items, answers, feedback ^ 
remediation, etc. The resulting practice items can then become ^^rt of ^ * 
either an on-line *or off-line course of ii^tructlon. The compSfer dialog 
can also be used off-line in the form of. 4 checklist .for the author, 

MONIFORMS were developed specifically- for, the PLATO IV system TUTOR 
language. However, t;he^oncep.t of programming templates which permit 
a,uthors'with limited programming experience to crfeate test and lesson 
materials has wider application. While MONIFORMS are a valuable first step, 
there still existed a need for more advancecl author aids. Preliminary study 
at HumRRO indicated the ■ feasibility of developing author aids which through 
-interrogation of the course author would automatically convert lesson content 
and structure into executable program code. Therefore, the author would 
require no "previous programriijLig experience and thus make th^ aids much - 
easier to use. This concept Wormed the basis^of the" approach for the current 
effort. . . . 



In summary witrr, respect to authoring aids: 

1. .There exists a very rich array of a wide variety of materials, 
handbooks, guides, and .automated aids which could serve to help in the' 
IPISD process. 

2. There exist? a very real need for these aids. 

* ■ 3. The selection of specific assistance to be integrat^^^' into t^e 
framework of IPIS^^ is a task that is yet to be completed. 

I 

EVALUATION 

In the previous sections, we have discussed the need" for specific 
aids or tools which can assist authors to apply the IPISD process in the 
preparation of instruction. In developing and tailoring aids for authors, 
we are in effect developing instruction— for authors. Once any instruc- 
tional product Is under development', a continuous process of evaluation and 



I? 



revision, need8 to take place. Nhis evaluat^v* proceBa ' ) 

aDDlled to the development author aids-an approach we have taken Itx . ..^ 
.the present; research effort. ' The following discussion of evaluation provides 
background and rationale for our approach tf evaluation. . . 

Evaluation Defined • . ' , •. ' , 

Evaluation is thb process' of delineating, obta^nltig, aid. providing 

useful iuformatior^in judging decision alternatives l^9l. It . . . 

related- process wl#ch has as it's ma^or characterl^stlc the dteterminablon of 
■-vaXue, woVth; or merit. The evWuallon process »ls conceived as- continuing • . 
• -mhe; than ^s Uavin«:,a; discrete be|ipning or ^'^^H' . ^^f "^'f ^^^^^^J", 

facilitate the qdntinuoua" improvement of a program, it should «^imulate, - . 

not stifle, instructional development. ; . « . , ' ; 

• Evaluation procedures may be categorized as formative or suron^atlve 
[701. Formative evaluation is that process which validates instruction 
during on-going initial program development. The results of this evalua- 
tion are acted u^^^Tl^ediately. in program modification. In other words, 
the practice of conducting tryouts of draft materials during progr^^ 
development, followed by measures whi^ provide an assessment of the 
materials wiich lead to their revisio'n is referred to as formative eval- 
"tion Fo5matl.ve evaluation is performed for the purpose of diagnosing 
■^nd^rrecting the weaknesses of a program. 



4 ^..^A An fhp. rpvision of instruction may be engaged in 

„„.sr:. c ;™ ;. »."™; ..... ». ..">. ms'" 

are constanf ly being evaluated ?^ S;alvatlo:i employing 

exists. However, it is a tormax pruaJ-**^ TPTQn r^mrt^RR and 

• various, assessment techniques which 1^ the keystone of ^^^^^"^^ B^"^ 
whLh provides the link between bourse content and course i'^P^^^^^"^* f 
expUcuIy stating objectives and criteria, one can ^rov^^ly /f l^^^^^- 
the program is achieving its goals, or if goals are- to be modified. 

Sumni^tive eval^ ion is performed for the purpose of assessing a 
fully implemented training program wl|^h respect to its ability to produce 
graduates who can perform to minimum standards. of performance. ^ Also , the 
Valuation can determine whether or not efficient and ^"^f ' 
made of educational resources. Summative evaluation should oc</ur after 
instructional development, improv^ent , and stabilization of operational , 
and administrative activities. This may vary from one training program 
to another. In some cases, training objectives may not be mea^urable^ 
at the desired time of evaluation because th^y are either too tostlyto 
measure or are long-terirt objectives. . /. 

Results of a summative evaluation, while of interest to the developer, 
' are of primary concern to th^se who will decide whether or -not a -Program 

is to be continued or adopted. Summative evaluation therefore, provides , 
the basis' for policy decisions that do riot necessarily concern revision 
of the program or product [71,721. 
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The differences between forjhatxLve and a(\innnative evaluations are mainly' 
in. their purposes and" the timing of ,t|ieir applifcatlon. Formative evaluation 
Is continuous liTna'tiwe and serves^^to refine k given program' through an 
lt4erative fteedftack process; summative evaluation produces final Judgment^ 
concerning th/ degree to which program objectives and goals have -been • - 
attai-ned. The information obtained from a summative evaluation Allows the 
uafei: to >3idge whether a program meets his needs, whether it should be widely 
drsserainated, and if alternatives exist,' which are to be prei^erred. 
\ ■ . . . - . / 

The discussion which follows will concentrate primarily upon- formajtlvd' 
evaluation, as ^he.purppse of the effort reported herein was- to evaluate" 
author aids Inl their i'liitial development stages/ , , - ' 

^Evaluation Mtfdels ^ • 

The formal distinction between formative and summative evaluation is 
^atUlbuted ko Scrivdn [70]. However,, thp purposes for which' such evaluation 
da4:a are ustfd have been discussed for m^y years in the training and 
education literature. Cronbach [73] stated that "the greater service 
evaluation can perform is to identify aspects of the course where revision 
is desirable." Early models of the systems' approach' to training development 
contain quality control components which emphasize the need for feedjjack 
for program improvement,. Smith [74] described thk purpose of a qMality 
tontrol system, 'A, ' i^^eans for continuous monitoring of the quality ■ 
of the graduates and for 'improving the training when It is deficient." 

Quality control procedures are needed both at the school and in the 
field. Information from both locations must be "fed back" so that the 
instructional program can be appropriately adjusted. Schools require two 
types of feedback Information. The first type assesses the ability of a • 
course graduate to perform acceptably i:hose tasks which tha instructional 
program claims to teach. This type of information assesses the ability of 
the instructional program to' teach well whatever it is that it claims to 
teaJih. ' ,In most instances, this "assessment- can be made at the school. 

A second type of feedback information deals with the discrepancies 
between the course graduate performances and f ield. i;equlrements . "Relevancy 
control" information assesses whether or not the instructional" program ■ 
teaches the appropriate subjects or tasks, and whether or not the student 
canVtranlsfer these capabilities to the field. Also, this feedback should 
prcJVxde information dealing with changing field requirements and with more 
precise descriptions of job activities. 

Baker and Alkin [75] point out that the evaluative process was an 
integral part of programmed instruction development which antedated the • 
surge of interest in formative evaluation during the past decade. 

Recentjjtuodels of the formative evaluative process include, those of 
Stake ^:'6] ,VSctiven [77], Stufflebeam, et al . [69], Sanders and Cunningham 
[78], and Rippey [79], Scriven [77] feels that it is best if formative 
evaluatior^VHLs performed by someone other than the developer. Scriven 
calls his approach "goal free evaluation" which calls for the evaluator 



. ... ,^ . . 

eo'-assess the actudl effects of the program. The eValuator operates with- 
out Wwledge of the purposes, goals, or objectives , of- th^ program 

- developers^' Another mode-L is described by Stake. [76l ae "responsive - 
evaluation" which call's- for the evaluator^ to be external, to an instructional 

.development activity, and therefore to have 4 certain independence and 

objectivity that i8'pi:;BSumed not to ))e pre&ent in an Internal evaluAtor: . 

The Stake model provides a process evaluation strategy, which contains a 
' two- St age procedure: the fills' determines congruence between what is 

4ntended-and what is actually, observed (that is^ discr.epancies ^from ^ 
Wrogram specif ipations) , andj Uie second with makirtg suye the program has - 

the type and quality of cgmponents implied by its objeotives. 

' ' ^ • ■ ■ * i 

Less dependence is placed on the external evaluator i)y Stuff lebe'am 

[69]. As this' model emphasizes the need for evaluation data to serve 

dec is ion -making purposes in a timely manner, it permits the evaluator to be 

part of the development team. The "process" evaluation component calls 

for provision of feedback continuously during program implementation. In 

a similar framework, Sanders and Cunningham [78] identffy four stages of 

the formative process. The first is called the predevelopmental stage, 

which seeks to identify needs. The sec6nd stage is called evaluation of 

objectives in which one develops, revises, and clarifies objectives. The 

third stage Is called interim evaluation, and seeks to evaluate each , piece 

of the instruction as it is developed. The final stage Is called product 

evaluation, in which the program as a whole is evaluated; after which It^ 

may be recycled for further development. 1 

■ • Churchman,' et al . [ 8O] discuss the question of whether to use internal 
or external formative evaluators. They make thf point (with which we agree) 
that in practice the formative evaluator will become 36 involved in the 
program that the objectivity expected from an external evaluator will be of 
little significance during the formative process. j 

Transactional Evaluation (Ri^>pey [79]; Seldel [81]) differs from other . 
evaluation models in that it focuses on the effects of. perception? of project 
team members and the user population. • Its usefulness in formative evalua- 
tion comes from its emphasis on making explicit the relationships, roles, 
problems and possible solutions as perceived by developers and potential 
users of the instruction. The formal involvpment of these .people in clari- 
fying the goals and objectives of a given pro-am contribute to Improvement 

- during its early formative stages, 

Formative Evaluation Techniques 

The same measurement techniques and procedures may be employed in 
formative and summative evaluation. It is the purpose to which the evalua- 
tive effort is put and the time when it occurs thdt distinguishes between ^ 
the two types of evaluations. 

The application of experimental design to evaluation problems conflicts 
with the principle that evaluation should facilitate t\ie continual improvement 
of a prograin. Experimental dfesign prevents rdther than promotes changes in 
the treatment because treatments canngt be altered in process if the data 
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about differences >)etweefl'-^t:rfeatnents are to be uneq'uii^^ocal [82]. The 
experlm^tal d^al^n type-;t>.f evaluation Is useful for making aummative 
decisions but almost uaojess as ^a device for making decisions during the 
plann:^ng and implement af'ion of r project [23^ 

In formative 'fevaluatlon, the developer is looking for what the 
researcher often takes grfeat pains to avoid. Instruction changes as a 
function o^ his actiyitry, bpth as it is being'develoRed dnd as it is 
implemented in pilot or field tests. The summafive evaludtor, on the 
othersHiand, as does the researcher, goes to great lengths to hold the 
progranvtonstant . , . • ' , 

■ . ' • ♦ • ■ .'• • • 

The choice of .design for a foriijative evaluation is a- com^plicated 
decision depending upon a number of considerations: cost, utility, 
practicality, fi-neralizability , etc. Campbell and Stanley [8i3] have 
discussed the major consideratidns in the choice of a design. The evaluator 
needs to be concerned with replicability In that if the ef fecj; of instruc- 
tion cannot be reliably established, then, of course, decisions about how 
to make it better are meaningless 7 

The most frequently used design in instructional evaluations is the 
single group pre-test/post-test design [84]. In this quasi-experimental 
design, a single group of students is first tested to determine how much 
of the criteria behavior they possess, then are administered the instruc- 
ticJn, then tested agairt. If learning gains are demonstrated, the product 
developer concludes he has a successful product. • The problem with such a 
design is th^t it allows many other plausible rival explanations for t'he 
observed results. In addition, a very serious l;I.mitatlpn is the unreli- 
ability of change scores [85]. 

^ ^- - s 

Pre- and post-testing is usually considered inadequate for formative ' 
purposes. Continuous monitoring permits correcting problems as they occur, 
tends to increase the aspects of the program that are included in the 
evaluation, and consequently improves the usefulness of the evaluation 
itself. 

One piy^lem with monitoring is in. collecting data representative of 
the performahce of the program such that it is typical of the full range 
of the intended usage of the system. This collection of performance iata 
needs to be done without disturbing the performance .of" the system being 
monitored, which is difficult. Another problem is assimilating and inter- 
preting the results. It *is easy to collect massive amounts of confusing 
data unless one establishes monitoring experiments with clear hypotheses in 
mind [ 86] . . 

The IPISD guidance [87] recommends such ttyouts, as follows: "If" 
the student who tries out the Instruction experiences difficulties, it may 
be profitable to again test out . the instruction, after revisions, on 
another student. Beyond practical considerations of time there is really 
no limit to the amount bf pre-test' tryouts that can be conducted until the 
instruction is successful." ^ 
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If the task of the formative ^valuator is to monitor programs in order 
to provide evaluation data leading to Improved instruction, then It is not 
surprising that the fodus of most research on formative evaluation has been 
at the data-acquisition/evaluatlon-utillzatlon juncture [75]., . 

' One interesting research question relates, to the selection of subjects 
as a data source for various formative evaluation efforts.. There -^re those 
(including IPISD) which recomirffend that formative evaluation data are obtainea 
from single learners in linear fashion with repeated tryouts. Essentially 
this technique 'consists of placing the author with a student as he/sh^ uses 
the i^iaterials. Ideally, the student will helg. the author ^/'"^^Kuities , 

efrots of sequence, and the like, and allow ;tlie al^or to test his assump- 
tions concerning tKe thinking processes which will4>e employed by students 
using the materials [78]. ' "V^ • 

An unpublished study by Robeck (as repSfted by feakcfr an Alkin [75]) 
•tested the feasibility of using a single Student as the data source for 
fdrmative evaluatlon,leading to the revision of an instructional P^^g^*™- " 
Thrstudy demonstrated that observation of a single student is an economical 
method for significantly improving instruction. Aside from this study, 
very lit tL rLearch on this technique has been performed; The present state 
of knowledge consists of a' number of conflicting "tips" on how to implement 
the- procedures. Some recommend that high ability students be used, others 
recoLended low ability. Some sources argue that students can only clean 
^rs-lantic and syntactic errors, while others insist ^^^^f^^.^^^^^^^f^^^,^" 
make more substantive suggestions concerning sequence, intended p^ereqUl . 
sites etc At present, even a simple experiment comparing the quality ot 
instructional products -which have and have not used individual student, 
tryouts as- part of the development has not yet been done. 

"as was stated earlier, a variet^ of techniques ca^ be usej for forma- 
tive evaluation. The purpose fot which the information is gathered determines 
whether it is formative or summative. ' The ultimate criterion of an instruc- 
tional program, however, is a change in the behavior of students. Determin- 
ation of whether or not that purpose was met requires a demonstration of _ 
such changes. The IPISD guidance states that one needs to examine in detail 
the responses of the learners on criterion tests [87]. A combination of 
tests observations, interviews, and affective measures is required to amass 
the data necessary for the formative evaluation and improvement of instruction 
The specific techniques and approach used in our project to evaluate the 
author aids will be discussed in Chapter IV of this report. 
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Chapter 11 
PURPOSE 



The purpose of the research effort described herein was to conduct a 
development and feasibility demonstration* of on-line, query-based author 
aids. The res^di^ch was designed to include author aids fot Blocks II. 2 
(Develop Tests) &nd III. 4 (Develop InstAiction) »of the • Interservlce 
Procedures for Instructional Systems Development (IPISD). 



Specifically, the activities of the project were to result in author 
aids which: ' * ' 

• Are suitable for creation of both on-line and. off-line instruction, 

• Are generalizable for differing subject matter areas. 

• Are documented in a flowchart form to permit timely conversion as 
apprtjpriate to other CAI systems. 

The utility of the author aids developed was to be evaluated and 
revised as necessary with military authors/instructors preparing 
operationally relevant instructional material. 
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" Chapter III ^ 
! APPROACH (3 

* The goal of the project Was to construct, implement and provide a 
feasibility test of on-line ai^thoring aidfc which can be integrated with 
the IPISD model. In order to jattain the objectives of this project the 
approach taken was: • j ^ , ' ' . • ■ 

• User-oriented I" 

• Guided by the IPISD moldel 

•*" Multi-level in its parallel development/evaluation activities. 

A cooperative working relationship was established with instructional 
and curriculum development personnel of the U.S. Army Engineer School 
(USAES), Ft. Belvoir, Virginia^ Input from USAES personnel was an important 
influence in the selection of ^uthor ^ds which would help the USAES 
instructional development team to implement the IPISD. 

Author aids to be developed in this project were presented on the , 
PLATO IV computer-assisted instruction (CAI) systeift.' During the course 
of the project, four PLATO IV terminals were located In the HumRRO 
laboratory, Alexandria, Virginia. In addition^ 8 terminals located at 
Ft. Belvoir, Virginia, were also available during the project. 

TheSsftgineer Non-Commissioned Office Advanced (ENCOA) course was 
selected for this project in cbnsultation with USAES curriculum development 
and training personnel and with the agreement of ARI. Arrangements were 
made to t>ermit participation of four instructors (2 NCOs and 2 Officers) 
who teach this course. 1 The ENCOA course covers a wide range of technical 
("hard") and soft skills. It was thought that if authoring aids ^ere 
developed which would be useful' for handling instruction and testing of hard 
skills (e.g., straightforward mechanical work) as well as soft skills such 
as' problem-solving, the set of authoring aids would be more applicable to 
other courses and other schools than if Just the h«rd skills were chosen 
for the targeted materials.^ Therefore, the subject matter selected for 
this project was a nine-hour block of instruction from the ENCOA course, 
covering such items as' field fortifications emplacement construction, 
U.s! /foreign mine warfare doctrine, and protective mining. The work 
involved in ^his secition of the course includes computational problem- - 
solvina, as well as procedural tasks. School personnel had defined training 
'ob4ectf5i<s as a result of previously applied systems engineering principles. 



■'"One Officer was transferred from the USAES during the course of the project 
Therefore, only three instructors participated in the research ^f fort . 



The IPISD model v^as aycompatlble and useful guide In designing the 
technical approach for the proposed project. Each of the first th«^e 
major pirocedural Phases— Analyze, Design, and Dfevelopr- were pertinent to 
the activities undertaken In this project. The targeted '^students" in 
this tfase were the authors and the Instructional focu s vas the author aid. 

■ ~ ' ■ 

The multl-^level nature ^ the project should be conslde-red here.- 
HumRRO personnel d6velope,d and evdluiited authot aids. Jhese author aids 
wer€r,i^Bi»n used by USAES Instructors to develop and validate instruction. 
Thus, iterative,* parallel activities occurred at different levels in the. 
project. Guiding all these efforts was the IPISD model^itself— in particular 
the first -three procedural Phases. For example, the' approach to author 
aid development and validation drew it^s guidance specifically from IPISD 
Blocks III. A and III. 5. T 

Initially,, a s6t o^ detailed flowcharts were constructed to describe 
Information elements arid features required by instructional- developers in 
performing the steps of IPISD Blocks II. 2 (Develop Tests) and III. 4 (Develop 
Instruction). The flowcharts were designed" to be sufficiently detailed 
and annotated for ready adaptation to any system (i.e., relatively 
hardware or software independent) . 

Inasmuch as the PLATO IV system yiaa considered a research vehicle 
only, care was t^en to maximize hardware or software independence of the 
aids. On-line author aids as well as off-line versions were . supplied to 
assist the author in preparing instructional and test materials for either 
CAI or non-CAI delivery of instruction. 

. The multi-level nature of the research activities is clearly demon- 
strated by the three levels of evaluation undertaken in the project. The 
first level was an Informal evaluation of existing IPISD guidance, procedures 
and author aids. HumRRO^s'taf f ', as users of this guidance, were the primary 
source of evaluation data at this level. 

Level 2 was directed toward a formative evaluatiorf of new author aids 
and procedures developed Specifically for on-line application to I?ISD 
Blo9ks II. 2 and III. 4. Level 3 evaluation assessed the adequacy of the 
instructional materials created by the military authors. These materials 
were then administered to U.S. Army Engineer School trainees who provided - 
an additional data sourqe. 

Revision activities occurred continuously throughout the period of 
project performance. The purpose of these revisions was to assure maximum 
utility of the flowcharts and author aids in implementing the IPISD process. 
The test items and lesson material were not revised as a basis of trainee 
data, because of tin^ limitations, but "these datia were incorporated as part 
of the research conclusion. ., * 

The project activities were divided into four major Tasks. These were: 



Task 1. Analysis and Determination of Required Author Aid Element 



■ • ; 

Task 2. Conyerslbn c^' Flowcharts to Interactive Propram 
Task 3. Ivaluatlon of the Programmed Materials 



Taek 4.. Revisions 



The activities and accotnpllshihfpts In e^ch of tHe 
the following Chapter. 
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ate njescrlbed In 
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- Chapter IV . 

. • ■ 

PROJECT ACTIVITIES AND ACCOMPLISHMENTS > 

TASK 1. ANALYSIS AND DETERMINATION OF REQUIRED AUTHOR AID ELEMENTS / ^ 

In Task 1 a detailed- set flowcharts was constructed which provide \* 
instructional system designars with the means of performing the procedures ■ 
called for by IPISD Blocks 11.2 (P^elop Tests) and III.4 (Develop 
Instruction). \ Activities during t4sk 1 were" conducted in two Phases: 
(1) expansion of IPISD flowcharts «p as to "provide greater detail of the 
specific activities reqiiired forAe'ath block (What To Do), and (2) selecting, 
identif^jing, designing and flowcharting of author aids for completing the 
activities (How To Do). ' " 

Develop What-To-Do Flowcharts /' ' 

The IPISD flpwcharts for Bioc|/s II. 2 and III. 4 shown below in Figures 
2 and 3 \j^re us^d as the basic frajnework for the HumRRO-developed flow- ' 
charts. The IFISD flowcharts provide a broad description and sequencing 
of necessary activities. However, "because of their global nature, they 
provide only minimal assistance to the instructional systems 'designer . 
Eac^ element of the IPISD flowcharts was expanded into detailed 
st5p-by-step sub-elements that must be performed (or considered) in 
Completing the specific flowchart block. With respect to the IPigi) activity, 
Develop^Tests (IPISD Block II. 2), the procedural steps described in the 
"Guidebook for Developing Criterion Referenced Tests" [58] were used 
heavily in the identification of the .sub-elements. Figure 4 is an example 
of how'f'one -such IPISD element, 2.6 (Determine Scoring Procedure) from 
Block' tl.2 was expanded into sub-elements. 

% 

.Jt was found that the activity descriptions shown in sub-elemetit 
blocks were not always sufficiently descriptive of the activities required 
by~^he block. Consequently, it was necessary to further flowchart several 
of these sjib-element blocks. An example of further flowcharting of sub- 
element bibck 2.6.1 (Determine Qualitative Scoring Procedures) is shown in 
Figure ,5. The blocks in italics refer to blocks already flowcha.rted in 
IPISD, All other blocks ai:e HUmRRO flowcharts. Blocks outlined in bold- 
face are t^locks for which author aids were developed in the project. A 
check mark ibove a block indicates that existing author aids have been 
Identified for that block. The narrative on the right of the flowchart 
further clarifies the block and lists any references to existing author 
aids. - . , , . 

The product which resulted from the Phase 1 activities is itself a 
valuable author aid for Instructional system designers. It provides a 
step-by-step enunciation of activities that must be performed. The revised 
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Quantitative 




QuaMtativo 



Oetarmint qualitativa icoiing 
procedures. 

2.6.1 



Determine quantitiative tcoring 
procedures 

2.6.2 



petarmine when tcoring will 
occur, after or during test. 



2.6.3 



betarmina whether scoring will 
he dona by hand or machine. 



* 2.6.4 



Will interference tcoring be used? . rs t i 

See Swezey & Pearlstern; Guidebook for Developing Criterion- , 

Referenced tests. * , , ^ « i. i i ^ 

Arlingtoh, VA: U.S. Army Research Institute for the Behavioral and 
Social Sciences. Auguat 1975. 



Develop form for recording correct and incorrect answers class 
and individuals. Develop method for determination of number of 
correct answers (or a scoring key), p. 6 6. 



Write scoring dir 


ections. 
2.6.5 






Perform scoring procedures 
tryout. 

Perform readability cl^ck. 

2.6.6 



Use on-line or off line readability aid. 
See Swezey & Pearlstein, p. 7-3 



Beviil procedures as indicated 
in tryout. 

2.6.7 
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and expanded flowcharts were produced as off-line materials. A? such, 
they can be converted to checklists and used as, procedural guide$ by 
designers qf instruction. » • , , 

It was not feasible witlvin the limits of this project to produce 
the fully expanded flowcharts on "the PLATO terminal. However, the 
information contained in many of the blocks was incorporated into the on- 
line author 'aids and as such provide on-line guidance in accomplishing the 
objectives of each block. 

Identify ftnd Reference How-To- Do-It Author Aids 

In Phase I the detailed activities (sub-elements) ne^ed for developing., 
tests and instruction were defined and arranged into sequential order. In 
Phase 2 each subrelement }^&s examined to determine, specific authoring aide 
desirable to accomplish the sub-element. In otjier words, Phas^, 1 describaa 
what must be done , ^d Phase 2 defines author aids for doing it . Time 
constraints did not permit the development or selection of author aids for 
every sub-element. Therefore, aids were provided for those sub-elements 
which were identified as of highest priority for potential users. The. 
selection-was based on such factors as: ^ " * 

• 1. Available HumRRO expertise gained from previous experience in 
author aid development; 

2. A review of the literature to identify «aids already available 
for use; and 

3\ Opinions of instructional systems designers at USAES concerning 
aids they considered would be helpful to them. 

After identifying the author aids needed for the IPISD sub-element 
blocks, the next step accomplished in Phase 2 was that of including 
references to the author aids in the flowcharts developed in Phase 1. The 
purpose of this step was to identify for users of the flowcharts those sub- 
elements for which author aids were available and to refer them to a 
reference (hardcopy or on-line) w^ich more ^ully detailed the specifications 
of the aid. » 

Throughout Phases 1 and 2, care was taken to assure that 'the flowcharts 
were sufficiently detailed and annotated to be of practical use to instruc- 
tional system designers .and would readily permit at^ptation to any system, 
i.e., be hardware or'software independent. 

Task 1 activities resulted in an Interim Report and Guid^ to the Use! 
of Flowcharts [88]. This report contains flowcharts providing detailed 
guidanc* on tht procedural steps necessary f9r implementing IPISD Blocks 
II'. 2 and 111.4^ and identifies sub-elements of these blocks for which author 
aids Were developed. In additioji, the Interim Report includes a guide to 
the usfe of flowcharts which we felt Is necessary inasmuch as many authors 
may not be familiar with a flowchart format. 
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The flowcharter prepared In Task 1 are of value from three standpoints: 
(1) They are useful as tools for instructional 83r8tein8 designers in the 
implementation of the IPISD process; (2) they may be used as a model for 
detailing the processes covered in other IPISD Blocks in terms of, level of 
.detail, style, and format; and (3) they may be used0.n the preparation of 
on-line author aids on any CAI system. ^ ^ 

TASK 2. CONVERSION OF FLOWCHARTS TO INTERACTIVE PROGRAM 

In Task 2, author aids Identified in Task 1 were developed for presen- 
tation on the PLATO IV computer-assisted instruction (CAI) system. Inasmuch 
83 the PLATO .IV system was considered a research vehicle dnly,.care was 
taken to insure that the author aids developed could be readily modified 
to be hardware or software independent. Where possible the author aids 
were also created so as to hAye application for "off-line" use. 

» 

In the context of the present project, an Instructional system developer 
(aut,hor) may be working on-line in an interactive mode with a computer.' In 
this cas^e, he is termed an "on-line author." If an author is i\ot working 
directly with a computer, he is referred to as an "off-line author." Even 
authors who are developing CAI materials work ii) both off-line and on--line 
modes* For example, some authors use preprinted CRT layout sheets to write 
their, text, and then have clerks key the text into the compute^:. Currently 
authors of most military instruction typically work off-line, although they may 
have access to computer support for such things as test scoring, statistical 
item analysis, or other aids. 

Author aid$ were developed to assist the on-line author in preparing 
instructional and test materials for both CAI and non-CAI delivery of 
instruction. CAI was the principal mode used in this research 'because the 
CAI mode provides-^ opportunity for ease of gatherin^^a^ analyzing data 
regarding both student and author activities. Off -line versions of these 
aids will assist off-line authors in preparing both CAI and non-CAI materials, 

PLATO lessons "inquiryl" and Vinquiry2" dan be thought of as master 
author aids. These lessons incorporate the individual author aids identified 
in Task 1. Lesion "inquiryl" deals with lesson aevelopmen t (IPISD Block 
iy.4), and "ijft^uiry2" with test development (IPISD Block 11.2) A Individual 
author aids^^r inquiryl and 2ican be roughly categorized in four different 
class^wWtW are discussed below* The four categories for each master 
author aid' are: 



^ • • • . ■ "\ 

These lessons wete available on the University of Illinois PLATO IV CAI 
systeni at the tim^ this effort was completed. 
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Lesson "inqulryl " (LeBSon Development) 

^ i 

1. Tutorial author aids for Lesson' Development 

2.. Author aids for management of the lesson •development process 

3. Author aids for lesson .content development ^ 

4. Author aids contrplling. within lesson branching 

Lesson "lnqulry2" (Test Deve lopment ) 

1. Tutorial author aids for Test Development 

2. Author aids for management of the test development process 

^ 3. Author aids for test development (e.g,» test instruction^ and * 
test items) 

4. Author aids for post-test reporting of results, review and 
remediation actions. ^ 

Less or^ Development (Lesson *'inquiryl") 

1. Tutorial Author Aids.' This series of author aids provide instruc- 
tional system designers (authors) with guidance in the preparation of lesson 
materials. On-line the guidance is automatically presented at appropriate 
points during the lesson development process • In addition to the forced 
presentation, authors may review any specific guideline as desired. The 
guidelines are , available in both on-line and off-line versions. Specif it 
guidelines included in this series of author aids^ are: 

• Instructional Seqqencing Rules » 

• Gdidellnes for the Preparation of Terminal Learning Objectives 

• Guidelines for Reducing Reading Difficulty Level 

• Guidelines for the Preparation of Text Material , 

• Guidelines for.thejUse of Practice Question Formats (General) • 

• Guidelines for the Preparation of Multiple-Choice Practice 
Questions 

- • Guidelines for the Preparation of True-False Practice Questions 
Guidelines for the Preparation of Constructed Response 
^ Practice Questions 

The "off-line" version of these author aids willvbe of use for all instruc- 
tional, modes. 

2. Author Aids for the Management of the Lesson Developoierit Process 

• Sequenaing of instHiation 

\ * / 

•* 

Instructional content is, of course, based on Terminal Learning 
Objectives CTLOs), Learning Objectives (LOs) , and Learning S^eps (LSs) • 
However, how these are sequenced In the instruction may very well determine 
whether an Instructional module is effective or Ineffective. (A module' 
fiB here defined begins with an LO or LS and Is usually followed by S-TO 
frames of text and practice questions which teach the LO or LS\) To 
assist authors in the creation of modules and the sequencing of Instruction, 
worksheets have been prepared for off-line creation of LOs, L$s, text 
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frames and various types of practice questions. These yorksheet.s can be 
used for most Instructional modes but are particularly suited for CAI and 
programmed Instruction. 

# Learning Objeotivee (LO) and Learning Steps (LS) 
Management 

Learning Objectives (LOs) and Learning Steps (LSs) ave the backbone 
the IPISD process. They dictate the content of both Instructlohal material 
and test Items. The Instructional system designer must attend to them 
carefully to Insure they are represented In the Instructllo^ and test 
situation. As described above (Sequencing of Instruction) each module of 
Instructlpn begins with an LO or LS. The author Is required to Input the 
LO or LS prior to Inputting Instruction for a given module. During the 
preparation of the Instruction for a module, the associated LO or LS Is 
available to the author as a continual reminder of the Instruction to be 
addressed. A by-product in the CAI version that is available to students 
studying a particular module of instruction is the option to access the 
LO or LS statement underlying the instructional module (see Student 
Controlled Branching Author Aid below.) The '*5ff-llne" version of this 
author aid will be of use for all Instructional modes. 

0 * Reading Difficulty Index " ' ' 

When preparing any instructional material it is essential for the 
author to consider the Intended audience for the material [89-97] r There- 
fore > an author aid was prepared for use on the PLATO system that auto- 
matically computes the rfeading difficulty of text material, question stems, 
and feedbacks provided the student as the material is inpirt:ted into the 
system. 1 In using the aid, authors specify the reading ability level of ' 
the intended audience and If this level is exceeded the computer so informs 
the author who can then revise the material to a lower reading J.eyel . An 
off-line version provides the formula, and identifies the components required 
for computing the reading difficulty index. Obv^usly, this author aid 
is far stronger in its on-line version since the author is not required to 
compute the ^ndex. However, it can be used manually' in off-line 
ii^structlonal modes. . 4 * 

3. Author Aids for Lesson Content Development 

0 Text' Creation and Editing 

' s 

Thi? author aid will be most powerful for development of CAI 
materials.- In CAI form it permits authors to create CAI executable textual 
material without a knowledge of the programming language required by the 
system. The text may be placed at the author's option any place on the 
screen and permits revision after initial creation. It incorporates other 
author aids such as the reading difficulty index (described eatJLier) . 



Available in Appendix A. 
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# PiHxotioe Item Creation 

. * 
This la actually a series of. several author alda which allows 

creation of practice questions. The author' aids do not require a knowledge 

by authors of a computer ptogr^iimilng language. The aids are of primary 

value In a CAI or PI mode. Detailed characteristics of each of these 

author aids are shown In Tables 1» 2 and 3. 

4. Author Aids Controlling Wi thin-Lesson Branching 

# Author -Dire at ea Branching 

Today, in many instructional modes such ps CAI, PI and other forms 
of self-paced instruction, students frequently must demonstrate a mastery 
of- current instruction before being allowed to go on to new instruction. 
Based on their performance, some students may be required to review certain 
portions of the instructional material while others will go through the 
instruction without forced review. That is, v student? are branched depend- 
ing upon their particular needs. This process requires that student 
performance be continuously monltorecl^. Author aids have been provided to 
assist authors in these efforts. Data collection aids for practice quegrtioris 
provide continuous student monitoring. Other aids prpvide guidance to the 
author on how to use monitoring information (e.g., the i^^umber of ^ittempts 
a student is permitted at a practice question, the copditions under which ^ 
the student is required' to review instructipn, etc.). These aids have 
primary application in self-paced modes of inatpuction. 

#' Student-Directed Branching ^ ' 

Stud|ents themselves, frequently know when they ne«d additional 
assistance and^^ould have the opportunity of accessing this assistance 
whenever they desire. However, they must be able to identify what assistance 
is available and the means for accessing it. This series of author aids 
provides students the options of accessing auxiliary information, returning 
to previously studied material^ and, if permitted by the author, of branching 
to the end-of-lesson test from anyplace in the lesson. TVhese aids make use 
of aids already developed for other purposes. " For example, the management 
and sequencing aids provide specific statements of TLOs, LOs and LSe asso- 
ciated with each instrj^ctional module. The student -directed branching 
aids permit the student to t>emporarily branch to these statements whenever 
desired. The author aids provided for student-directed branching will have 
wide application independent of the instructional mode used. 

Test Deve lopment (Lesson "inquiryZ") 

V 

1. Tutorial Author Aids. Thl,s series of author aid? is similar to 
those discussed In the tutorial author aids for lesson development. These 
aids, however, provide both on-line and off-line guidance i(i the various 
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That is, back-page, return to beginning of lesson or beginning of module. 
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Table 1. CHARACTERISTICS OF PRACTICE QUESTION AUTHOR AID 
^ (MULTIPLE CHOICE). 



• Question stem has maximum length of six 50-character lines. 

• Three-six answer alternatives (including (iorrect answer) permitted, 

J 

• Each answer alternative has ftiaxlmura length of two AO-character 
lines. * 

• Correct answer position ratidomly selected. (Author may select, 
other position If desired.) 

• Author "cued" If answer alternatives differ In J^ength by more than 
+ 20 characters. (Author has Option of revising.) 

• Author specifies one-t^ee attempts student permitted on question. 

Author can create correct answer congratulatory message. . Maximum 
of five 40*-character lines. 

• Incorrect answer feedback messages may be specific to response 
given, or general feedbacks which may be different for different 
Attempts. IncoVrect answer feedbacks are limited to five 40- 
character lines. 



• Correct answer given student If number of permitted attempts 
reached without student correctly answering the question. 

• Reading difficulty of question stem and feedback messages 
automatically computed. If deslr^^d reading level exceeded, 
author has the option to revise mAerial. 
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Table 2. CHARACTERISTICS OF PRACTICE QUESTION AUTHOR AID 
(TRUE-FALSE) , 



m Question may be 'a maxlimim of six 50-cKaracter lines • 

• Author specifies one or two attempts student permitted on question., 

• Author can create porrect answet congratulatory message. Maximum 
of five 40-character lines. 

• Incorrect answer feedback message provided if two attempts 
permitted. Messages may be a maximum of five 40-character lines. 

• Correct answer given student If number of permitted attempts 
reached without student correctly answering question, 

• Reading difficulty question and feedback messages automatically 
computed. If desired reading level exceeded, author Tias the option 
to revise material. 
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Table 3. CHARACTERISTICS OF PRACTICE QUESTION AUTHOR AID 
(CONSTRUCTED REPONSE) 



• . Question may be a maximum of six 50-character lines. 

• Author specifies onarthree attempts student- permitted on question. 

• Student response analyzed for one--four correct or partially 
correct answers. 

Student response analyzed for one-four anticipated incfcrrect 
answers. . 

0 One--four Congratulatory messages permitted depending upon number 
of correct or partially correct answers specified by author. 
Messages may be a maximum of* five 40-character lines. 

• One-four wrong answer messages permitted depending upon number 
of Incorrect answers specified. Messages may be a maximum of 
five 40-character lines. 

• Author hks option of permitting misspelling of answer; words in 
answer to be out of order; extra words in answer; and disregarding 
the capitalization of answer. - ' ^ 

• Correct answer given student if number of permitted attempts 
reached without student corre,ctly answering^ the question. 

m Reading difficulty of question and feedback med^ages automatically 
computed. If desired reading level exceeded, author has option 
to revise material. 
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facets of test development. Like the leason development tutorial author - 
aids, they are biitomatically presented to authors at appropriate points 
during the test development process and are available for review at any \ 
time. Specific guidelines included in this series of author aids are. 

\- ■■ • .. 

• Guidelines fof using "inqulry2" author aids 

• Guidelines for the Preparation of Terminal Learning 
Objectives 

• Guidelines for Writing Test Instructions 

• Guidelines for Reducing Reading Difficulty Level , 

• Guidelines for the Preparation of Multiple-Choice Test 
Items 

Guidelines for the Preparation of True-False Test Items 

Guidelines for the Preparati^ of Constructed Response 

Test Items v ' ▼ 

Guidelines for Assigning' Scores to Test Items 
Guidelines for Post-Test student review of test items 
and Remediation Strategies ^ - , 

2. Author Aids for Management of the Test Development Process 

0 Sequencing of Test Items ^ ^„ " ' ' - 

Worksheets are provided to authors for off -Hue creation of 
TLOs, LOs, L& test instructions, and test items. These aids permit author? 
to organize and sequence their t^st Items prior to. input Into the computer. 
The worksheets are useful for all modes of lnstru9tlonal delivery. 

0 Terminal Learning Object-ive'e (TLO), Learning Objeatives, (LO)^ 
and Learning Step (LS) Management 

This author aid is somewhat different from the corresponding aid 
used for creating TLOs, etc., in the lesson development process. Instruc- 
tional system designers (authors) input all TLOs, etc., into the computer 
in the sequence in which they wish to cover them in the test. (See sequencing 
qf Test Items above.) The author aid then maintains records of which TLO, 
etc., has been addressed in the te&t and in the on-line version. The 



computer 
^f*-iine 
modes . 



"cues" the author as to the TLO they- Should next address.. 



The 




'Sf^^iine version of this author aid will be of use for all Instructional 



'.ading Difficulty Index 



This author ^id is identical to the' one discussed earlier In the 
Author Aids for Management of the Lesson Development Process. 

3. Author Aids for Test Development. 

# Creation of Test Inetruatione and: Editing 

This author aid is similar to the text *treatlon author aid 
previously described. The aid will be most useful for development of 
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student Instruct ion» for computer-administered testa* It can also be used 
for on-line development of instructions for other Instructional modes such 
as programmed texts, etc. The use Qf the author aid does not require a 
knowledge^ of the programming language required by the system. The aid also 

permits revisions to be made to the instructions after initial creation. 

•n ■ 

# Test Item Creation . 

A series of author aitfs was developed to be used for creation 
of representative types of multiple-choice, constructed response ^and true- 
false test items. They are similar to the practice question author aids 
(see Tables 1, 2 and 3) except, th^t tl?ey cW not provide correct |and 
incorrect resppnse feedbacks, nor a variable number of perinitt^itt attempts* 

# Timing of the Teet * . ' 

A 

\ 

This author aid permits authors to establish ,v If desired » a time 
limit for individual items in the test or a time limit for the^entlre test. 
In the CAI version of this author aid, the co mpute r maintains a record of 
elapsed time and ^kes appropriate action Ua^^Ud uijHLtje elapsed time. This* 
author aid is most'^useful in a CAI mode. ^ 

# Test Item Scoring ^ 

Test item scoring author aids are provided to assist test de^i^elopers 
in establishing test scoring procedures. These aids include such consider- 
ations as: setting cut-off scores, differential weighting of various 
answers to a test Item (i.e., correct, partially correct, and incorrect 
answers), and/or differential weighting of different test item?/ Off-line 
version^ of these aids consist of guides; checklists, etc. On-line versions 
are similar but are prepared in a ^'query" format. The aids are useful for 
test scoring for most instritctional modes. 

^ I, 
4. Author Aids for )?eporting of Results, Review and Remediation 
Actions ^ 

.. ' ' * 
. # Repovtmg of ' Results 

Subsequent to test item scoring (discussed above) authors can - 
establish the minj^mum passing score required. This author aid then scores 
the test and automatically reports to students their obtained score and 
the minimum score required for passing: The aid is most ^werful In a 
CAI format but may be also used In other self-paced modes of instruction. . 

# Post-Test Review of Test Items Missed 

These author aids permit the test developer different student 
review options for test items missed. For example,- if ^the student passes 
the test with less than a perfect score, the author may elect ^o show the 
student the correct answer to items missed. . Or, in the case of students 
who fail the test,, the author may elect to: (1) show the students test 
items missed without providing the correct answers, (2) show fest.itero^ 





8ed-^n<i i*i|(p^lude the correct answers, or ^3) not permit a review of 
mipsedV' ;Thi8 author aid is primarily of use in a CAI or programmed 
mode« , .\ * * ' .* 

,„,:■ ; ■ ••■ . ■ . . 

# Ji^mediation Actione 

■ \\ - . ■ ■ ■ 

^ This lis' a series of author aids which permit the author to select 

the T^pe of\.acti^n that will be taken if a student fails the test. THe 
acti*d|^ Possible W these aids are as follows: 

^>j^>>u^^ Rdr-atlminlstration of instructional li 



hv«»/ ivipr-aMuixujLDui.ciuxi^ii kjx, j.uo ui. u xuuao: lesson followed by re-- 
admin^.j^ra^ of ^^eat items previously missed. 



Re-viiOTiinistratlbn of instructional lesson followed by re- 



administration of ^rimdre test, 

\ (c) Immedii^fce re-^administra'tion of test items previously missed, 
(No re--admj|ill;ist^ lesson.) ^ 

im^diate re.-administration of entire/test. (No re- 
admin i strait ibn^v^^ instructional lesson.) 




C^)V^Re* of test Items missed. Give student 

option of revi)ro|a?^^ instruraiional les'son 'first , 

(f) F^^^dnjiiaistrat^pn of entire test. Give student option of 
reviewing instru<?t;J^nal lessoi^^ first * 



(g) , No r^u7administration of instructional lesson or test — 
student is fiilished Viiith iVsson or^goes to new lesson. 

In the on-line Vers^top' pf these author aid^, failing students are ayto- 
matically branched JaS'jiifec.t^^ bj^ the author. Therefore, these aids are 
most powerful in ^;CwW^ format . ^w^ver, 'the principles underlying the 
aids can ^ employed iivY^any instructional mode. 

TASK 3. EVALUATION " ■ H 

Three levels of evaluation were undertaken in this project. The- first 
level was an informal evaluation of existing IPISD guidance, procedures and 
author aids. Six HumRRO personnel with technical expertise in systems 
engineering procedures judged the ease and effectiveness with which selected 
IPISD procedures and guidelines could be used to develop instruction. Where" 
appropriate, these author aids were re.ferencecj, in the flowchart! developed 
In Task 1. 

The second |Level, of evaluation was a comprehensive formative evaluation 
of the new author aids aiyi procedures developed for ^pplljcatlon to IPISD 
Blocks II. 2 and III. 4. Three instructors from the ENCOA cotirse [.one officer 
and two NCOs (E8) ] served as study participants and as a data source for 
evaluation. These instructors used the aids to create test items and lesson 
;mat^ri1al. ^^aXuation data wete gathered a^ the authorar developed their 

^36 ■ !. 
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Instructional material. ^Formative evaluation of the author aids was 
accomplished by examining author performance and acceptance of the 
aids. Crlttcfl* weaknesses In the aids (l*e.> those which Impeded the 
progress of the authors) were reqiedled Immediately upon diagnosis of the 
problems* - ^ . 

In the third, evaluation level, the adequacy of the Instruction created 
by the military authors was assessed. This Instruction w^s administered to 
'U.S. Army Engineer Schodl trainees who provided the data for evaluation. 
The ultimate criterion of instruction is evidence of desired changes in 
trainee behavior (l*e., Does It "teach?"). In order for t\ie author's devel-- 
opmental activities to be adi^quately assepsed , ,t ralneef performance and 
attitude data were collected. 

In the intlal stages of the second level formative evaluation, HumRRO 
staff functioned as "test item developers" and/or "prepc^rers of lesson 
^at^rial." Their role in this atudy was to find errors and faults in t;he 
direct Ibhs, requirements, procedures,^ etc .4^ of the' author aids. We t\jpen 

used these '^data to make needed revisions of the author aids. v. 

» ■ .. . 

Once the author aids were considered ready for application to actual 
course content, the three USAES authors were given training in using the 
aids. A brief 15-minM|:e famlliarizatdLon/ training period in using PLATO 
preceded each individual's involvement in the project. They received 
instruction in signing on and off to the system (which Included signing 
into the appropriate «HufnRftO iesson) . A brief (approximately 5 minutes) 
orientation to the PLATO keyboard was then presented to each author. This , 
Included: use of the edit keys; editing techniques; and use of thft help V 
sequence keys (e.g., HELP, BACK, NEXT, etc.). Descriptions of system 
crashes, transmission errors, and other system abnormalities were provided 
along with instructions on how to proceed, un^er these circufostances. The 
authors vtere then permitted to practice with the keyboard before they 
started inputting their lesson/ text materials, and ail of them chose to do 
so. 

* *■ . • 

Following familiarization training 0*31 PLATO, the three USAES instructors/ 
authors were given a brief explanation of their role a? authors and then 
training in the use of the author aids* Project staff members provided the , 
training in a one-on-one, tutorial mo(3e. 

Following training in the use of aids, Instructors prepared and input 
on-line in the PLATO system test items and lesson material. Each of the 
Instructors developed a lesson and the related test items in their content 
specialty as part of a 2-3 hour block of different, but related, subject 
matter from the Englneei; NCO Advanced (ENCOA) course. Table 4 lists the 
subject-matter blocks selected foi^ this project. 

The ENCOA course had undergone systems engineering and USAES personnel 
provided a set of well-defined terminal learning objectives. Test items 
were prepared which reflected these objectives. An additional advantage to 
the ENCOA course was that both NCO and Officer Instructors were available as 
authors. Henpe, the utility of the aids could be evaluated across a wide 
range of background skills and experience.^' 



Table 4. EKCOA LESSONS AUTHORED ON PLAT,0 USING INQUIRY AIDS 



Lesson 

Field Fortlflcatlpna 

Emplacement 

Construction 

US /Foreign Mine 
Warfare Doctrine 

Protective Mining 

- TOTAL 



Completion 

"> Time to Time in 

Authoif?y Create Lesson Current Course 



Officer 



NCO-2 



48,5 hrs 



41 hrs: 



35 hrs 



124 hrs. 



hrs. 



2 hrs'. 



3 hrs. 



9 hrs. 



Average 
Completion Time 
in CAI Version 
(minutes) 

94.3 (N " 9) 



60.2 (N - 11) 



38. 6. (N - 9) 

191.1 minutes 
<or) 
3.22 hrs. 
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Although^ach aut;hor f^as required to input his material into the 
computer, a HumRRO staff member was present to^asslBt him in the process 
on a one-to-'One basis. No instructor needed to know the TUTOR language or 
have previous TUTOR^ejtpetience because the INQUIRY author aids were designed 
so that code was automatically generated. ' ? • 

N , ^ " 

rData collection, to a large extent, resulted from direct observations 
of the authors creating and inputting their instruction and from structured 
interviews with the authors. We gathered user acceptance data, and informa- 
tion on various areas of difficulty that the authors experienced while using 
the aids. ' 

The authors using the INQUIRY system were encouraged to comment at a^y 
time on their progress. Monitors were present at every inputting session 
to note any prpblems encountered or comments made by the authors. These 
comments were used later to make changes in the system so that it was 
easier to use. After the authors Inputted all of their material, they 
received a questionnaire asking their opinions of CAI and the INQUIRY system 
In addition to, Interview/questionnaire data, performance data were collected 
Such items as the time to create a given frame of text or test item on-*llne, 
the number of times a piece of text had to be re-lnpyt, errors in attempts 
to apply a particular aid, calls for help from the monitor, etc., were 
recorded. 
»\ 

In* the third evaluation level, weC assessed the Instruction created by 
the authors using the INQUIRY aids. To the extent feasible, studeot- 
identified areas of difficulty in the .instruct ion were associated with the 
U8fe of particular author aids. Jn this way, we tried to determine whether a 
poorly designed aid Ted to unclear Instruction or to problems with the tests 

1 V 

Twenty--two students wettt through the lesson material for about 2-3 
hours each to assess the quality of the instruction created with the author 
aids. All students received preliminary training on Using PLATO. As all 
the students o^uld not go through all the Instruction and testing within 
the time allotted by USAES for this project, only two of the three lessons 
were presented on a random basis to each student. 

Presentation of lesson content occurred on-line, as did the administra- 
tion of the post-tfe«ts based on the TLOs. In addition to collecting 
cognitive data regarding student performance, exit questionnaires were 
administered to obtain information regarding opinions of the clarity of 
the instructional material, problems encountered in the practice and test 
items, and attitudes toward the CAI instructional experience. 



1 

Prerequisites for selecting students were that they be NCOs who have entry 
qualifications for the ENCOA course, but have not been exposed to the 
material covered in these lessons. 



Findings 

1 . .Instruction and Test Development . The tutorial aids were presented 
to each author prior to lnp\ittlng. In no Instance did the author? seek to 
reread these aids which presented guidance on test and lesson development. ' 
Thus» It cannot be concluded whether or not these guidelines were useful to 
the authors. It ai^pears that more emphasis on the applicability and vatue 
of the aids Is required In order for authors to pay attention to this 
guidance. This may Involve a considerable change In their presentation 
fc^rmat. * . 

As a result Qf the Initial formative evaluation, the authors were 
able to prepare test and lesson material with minimal difficulty. In 
developing almost ^360" frames of instruction and testing, jbl total of 65 
problems were experienced by the authors dS' recorded by the monitors. Over 
25% (18) of these problems were trivial errors caused by the author pushing 
"he wrong key. Fourteen ins^^nces were due to unclear INQUIRY instructions, 
which were remedied as soon as/ "fVossible after they were noted. Twenty-one ^ 
problems were due to "bug's*^ in the ^INQUIRY program which were eliminated as' 
soon^as their diagnosis was conf'lr:med. Six problems were noted as due to 
PLATO system crashes and transmission errors. Six other problems arose 
j^rom miscellaneous jeasons. Thus, most of the instructional and test devel-- 
opment activity unde^rtaken by the authors occurred smoothly and without 
undue difficulty. 

2- Time . The blocks of instruction from the ENCOA course which were 
put on-line are traditionally taught in 9 hours. The average completion 
time wad under 3 1/4 hour^e for this instruction including taking the 
associated tests. (See Table 4) 

■V 

The time to prepare the test items and lesson materials using- the 
INQUIRV system of author aids varied little from one Author to cj^nother. 
Times ranged from 35 hours for one NCO (15 hours on-line), 41 hours for 
the Officer (14.5 hours on-line), to 48.5 for the rsfnaining NCO (18.5 hours 
on-line) . ; 

3. Readability Index , fhls , author aid provided information if the 
reading grade level was surpassed for each text frame or test item. How-- 
evter, it was, rarely used. That is, no matter what the index showed, the 
authors chose to ignore it\ About 220 text frames of Instructloii were 
produced in this study. More then 50% (126) exceeded the pre-specif led 
reading levels. However, only jL frame >Qf instruction was revised by the 
author as a result of this information. This was most likely due either 
to a lack of confidence in the measure V validity, or to a lack of percep- 
tion on the part of the authors regarding the crltlcallty of reading level, 
or a combination of both. *ln any event, no changes have been made to thls^ 
aid yet. However, we believe that there are at least two possible changes 
needed. First, authors should be given more instruction in the usefulness 
of this aid together with more practice. Second, the options available in 
the INQUIRY system to override this aid should be removed entirely or severely 
constrained (i.e., within 1 grade level on either side of the pre-specif ied 
orle) . , * ' 
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4. Mu1t1p1e|"Cho1C€ Items . As ^authors prepared multiples-choice 
pi^actlce and te^t items, a pre-programmed INQUIRY aid assigned the correct 
answer alternative on a random basis* Authors were given the option to. 
change the designation of the correct answer alternative, and approximately 
half the time exercised thla.optlon» Authors were/, thus. Indicating* their 
preference for retaining control over the manner in which they created 

instruction. ' . - 

- V 

Another author aid compared the lengths of answer alternatives and 
indicated when they were unequal. This occurred in about half the items. 
However, author^ unanimously disregarded this information and left the 
alternatives as they were. It appears that mdre restrictions on the author 
aid are needed In order for these. author aids to be used. 

5. Constructed Response Items . in the constructed response format, ^ 
authors used the follow^lng aids: 

*^ 

• The aid which permitted them to define the rigor with which 
answers would be scored. Authors selected those options which 'permitted 
misspellings, extra words, and optional capitalization. However", authors 
did not permit the words in the answer to be out of order. 

• Authors made full use of the vkr^pus aids available for 
preparing response feedbacks and varied betwefep providing trainees 
specific as well as ^neral feedbacks to both anticipated and unanticipated 
answers. The most positive reaction by dtudents was to the explanatory 
feedbacks presented after each response to practice questions. The 
INQUIRY author aids for presenting response feedbacks were used frequently 
by the authors and, if possible, should be Incorporated in off-line 
instruction (e. g. , using the guidance for preparing feedbacks in PI texts). 

• Authors were able to use the INQUIRY aids to specify anticipated 
correct and Incorrect answers. However, there appeared to be a problem 
with anticipating all the answer^s which were given by the trainees. 

The student attitude questionnaire data indicated a strong negative reaction 
to the constructed response questions provided by all tKe authors both 
as practice and as test items* Student performance data supported this 
result, as most difficulties were encountered when responding to constructed 
response questions (both during learning and test taking). These results 
appear to be due to those Instances iii. which a "correct" answer as given by 
the student is considered to be incorrect by the system. 

The monitors had observed this problem as authors input their 
material. The authors could not adequately anticipate all the synonymous 
correct answers' which could be given by the trainees. This problem is 
particularly critical in CAI, as the eyaluator "knows" whether the answer 
is correct after seeing it. This finding suggests that the guidance for 
preparing constructed response items be revised to clarify situations where 
authors should or should pbt use constructed response questions* That is, 
constructed response fonrfats should be used only in cases where the number 
of possible alternate correct answers is small* 

.1 



6. Student Options , of the student optiona, the ability to back up 
to a previous frame (BACK) was considered helpful by- almost all^of the 
trainees in all three lessons. The other three options were: HELP—ln 
which the relevant learning objective was displayed; LAB— In which the 
student could return to the beginning of- the lesson; and DATA— In which 
the student could go back to the beginning of the module. All three 
options were rarely used, and so it was not surprising that students were 
divided In their opinions about their necessity. 

TASK 4. REVISIONS . 

Task 4 activities consisted of making revisions to: (1) flowcharts 
developed in Task 1 and (i) author aids provided In Task 2. The purpose 
of the revisions was to assure maximum utility of the flowcharts and author 
aids in Implementing the IPISD pVocess. Revisions constituted a series 
of activities which spanned almost the entire research period and paralleled 
all development and evaluation actions in the other Tasks.^ Information 
sources upon which revisions were based are as follows: 

• As flowcharts were being developed. HumRRO personnel not directly 
involved in the project provided, input as to t^e clarity, utility and peed 
for revision. , ' 

• The expanded flowcharts developed in Task 1 were submitted to 
instructional syetem designers at the USAES for review. 

" Review of the Interim Report (which contains flowcharts) by the 

COTR provided additional information for needed revisions. 

• As on-line author aids were developed they were initially used by 
HumRRO personnel to identify "bugs" in the aids which were corrected 
before wilder use was made of them. 

• The most important test of the utility of the flowcharts and Author 
aids occurred in Task 3 when authors participating in the research effort 
used the flowcharts and author aids for developing Instructional material. 
Only minor "bugs" were identified at this stage since the flowcharts and 
author aids had undergope extensive review and pre-testing. Any problems 

" encountered by the authors in using the flowcharts and author aids were 
Immediately corrected. 

• The last information source for flowchart and author aid rpyision 
was to occur after the students had been administered the instructional 
materials developed by the authors. A few such needed revisloAs^were 
identified as a result of difficulties students had with, the ingtruction 
that was directly connected with the author aid used for preparing the 
instruction. Specifically, it was found that the author aids for preparing 
constructed response practice and test Items require additional develop- 
mental effort in guiding authors in the identification of what constitutes 
a correct or Incorrect answer. For example, the answer to a question might 
be 820 meters. However, if the student answered 820 M (which should be. an 
acceptable answer), they were Judged as having given an incorrect response. 

42 <J8 



As a retylt of the Input received from USAES •ys.tem deslgnere and th 

COTR, major formatting revlalona Were made to the flowchart* contained In 

the Interim Report [88] . The revised. flowcharts are shown in Appendices 
B 4nd C. 
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Chapter V 
RECOMMENDATIONS 



, Th^ evaluation of the author aids reported in the preceding chapter 
has demonstrated the feasibility of on-line aids for implementing IPISD 
Blocks II. 2 (Develop Tests) and III. 4 (Develop Instruction). User accep- 
tance of the aids is high and the time required for creation of test and 
lesson material has been significantly reduced. Further developmental 
effort of on-line author aids appctars warranted. Continued development 
effort should Include five major aress which are discussed below. These 
areas are: 

1. Modification of selected current author aids developed in the 
present project. » 

2. Development of additional author aids for IPISD Blocks II. 2 - 
and III. 4. 

3. Development of author aids for other blocks of the IPISD model. 

,1 

4. Conversion of aids presently programmed for PLATO IV CAI to 
oth^r systems . ' 

5. Author Characteristics. 



MODIFICATION OF CURRENT AUTHOR AIDS ^ 

There was insufficient time during the project tov make all of the 
modifications that were indicated during fomative evaluation. These 
modifications should be made if the lesson and 'test development author 
aids are to be maximally -effective . The specific author aids for which 
we recommen^ .modification arer 

(1) Reading Difficulty Index . As was reported in t^e previous 
chapter, experimental authors did not revise lesson or test material when 
the material was written at a reading difficulty level in excess of that 
Intended. Hence, if reading level is critical, the author aid should be 
modified to force authors to revise material when the reading level is 
more than one grade above that desired. 

(2) Author Aid for Creating Constructed Response Questions . Au^thors 

require additional guidance ^.n determining how to use construfcted response 
questions appropriately, yi^en constructed response questions are used, 
guidance is needed in the selection of the correct answers and alternate 
forms of the correct answer (e.g., George Washington, Geo. Washington, 

t 



President George Washington, etc^ ) . The need for this modification arises 
from the level 3 evaluation. Students who were administered the experimental 
V ^""^^^ difficulty in answering constructed response questions. 

^ Often, students would provide answers which were actvialXy correct but with 
an answer yarlatloh not anticipated by the author. Therefore, their answer 
Wbuld be erroneously judged incorrect. When this happens, it is very 
frustrating for the student and if it occurs "frequently, it reduces the 
perceived instructional value of the lesson tg the student; To remedy this 
. eltuatlon , authors should be provided with detailed guidance on the use of 
constructed response questions as well as guldante on the framing of correct 
answer variations. . . * ^i^hcul 

Editinfl of Test and Lesson Material, with the present author aids 
• ? must occur onl^ during the creation of text or questions. Once 

material has beeh completed there is no provision for further editing 
This is a severe weakness of the present author aids. It is possible to • 
revise the author aids so as to permit text and question revision after 
trial administration of the lesson. However, this is a major effort out- 
side the scope of the current project. 

.f > ■ ' 

ADDITIONAL AUTHOR AID^ FOR' IPISD BLOCKS II. 2 AND til. 4' 

Although additional. author aids could^have been developed ,' this was 
outside the current scope of work. Additional aids which are desirable 
Include: 

* AH „. ^ 

f ' ■ ' 

• Author Aid for creation of Matching Questions. (This aid is 
presently in draft form.) ■ - 

• Author Aid for creation of Arithmetic Manipulation Qu'estions. 

• Author Aid for creation of Multiple gholce- Questions with more 
than one correct answer. 

CONVERSION -0F INQUIRY AUTHORS A.IDS TO A CAL SYSTEM OTHER THAN PLATO IV 

.As previously stated, the PLATO IV CAi system was considered to be a 
research vehicle only. The goal was to develop and .document on-line" 
author aids that could be progranped on any CAI system. Therefore, a trial 
conversion of at least selected author aids should be undertaken.- This 
undertaking would determine the extent, to which author aids developed on 
aT^.^ .^V^^'^ ^r^"^ converted to another CAI system and point out 
difficulties to be expected in sUch a conversion. For example, rather than 
using the TUTOR language, a system-independent language such as PLANIT ► 
could be used as a test, for the general usefulness of the on-line author 
ri . /t flowcharts ^which supported on-line "aid development on PLATO 
IV could be used for,, the basis for this effort. . ^ ' 
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DEVELOPMENT OF AUTHOR AIDS FOR IPISD BLOCKS OTHER THAN 11.2 AND 111.4 

the present r^l^-J^^^^-J^XmU^^^^ ' 
on-line. wSlch is directed toward providing on- 

fff^rts should undertaken fot other IPISD blocks. 

AUTHOR CHARACTERISTICS 

Authoring, of CAI lessons requires a certain di^^^^^^ lev^-/ 
c6Hipetet>c^ which my not be P^f^^^^i^^f^i^'^re author Tch'n^re thSn was 
task. Aids are thus '-f "J^We an^ ^PPl^^^ 

done in INQUIRY, in ^/^^^ .^^"^"^.f IJ^?^! ^rerequlsi^ foV authoring , 
in creating effective ^-^^-/.f^;,/^: fe e'ab'lish.d as well as the extent 
both qn- and off-line "^^^^1^%^ J^llZ. experience between personnel, 
to which aids can compensate for ^^^j^^^^.^^^f ^^"ficiencies arte lacking 
If s.uch a study indic^ates that "^^^y/^^^^^^-^^f 3^J°eaon and classification 
and cannot be overcome by ««^^°^/^^^^*^^:^3:3::enrof '' author" Job/duty 
problem would have been uncovered and an assessmen^^ 
position requirements. is necessitated. ' . - 
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Appendix A 

*- . i r ■ 

READAtlLITY LEVEL FORMULA 

Figure the average length of a sentence In number of words. Figure the 
average word length Iq number of letters. 

1. Multiply the average sentence l^ength by .5. 
^ 2. Multiply, the average word length by 4.71.. 

3. yVdd the products of Steps 1 and 2 together. ' 

4. Subtract 21.43 from the sum obtained In Step 3. This fs the 
readability level of the j^terials. 

'VI 

Here is the formula: ^ • .^ ' 

r I 

[ (.5 (veralge sentence length)) + (4,?1 (average word length)) ] - 21.43 

(from Kincald, 1972) 



•1^ 
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Appendix B 



FLOWCHARTS OF IPISD BLOCK 1 1. 2 (DEVELOP TESTS) 
mo BLOCK in. 4 (DEVELOP INSTRUCTION) 



GUIDE TO THE USE OF FLOWCHARTS 

The* IPI^D flowcharts for Blocks 11^2 and III. 4 (aee Figures 2 and 3 
in Chapter IV) were used as the. basic framework for the HumRRO-deve loped 
flov/charts. The flowcUarts in this appendix expand each of the IPISD 
flowchart blocks into detailed step-by-step components which must be per- 
formed (or considered) in complecting the specific flowchart block. (In 
th^ appendix IPISD flowchart blocks are noted by italics.) 

Flowchart blocks which are shaded ate blocks for which author aids 
were developed in this project. Next to these shaded blocks are Indica-- 
tors specifying whether the aid is on-llr>e and/or off-line. Blocks with 
an asterisk (*) next to them indicate that Existing author aids have been 
identified. In these cases we provide a reference to th^ author aid that 
is to be used at that specific point in the process of developing tests 
and. instruction. For some flowchart blocks supplementary information is 
presented for clarification of a specific block's activity statement. 

For example, Figure H-1 showg the first seven task elements required 
to prepare multiple-choice test items. The tot^ task elements can be 
found on pages B-7 thru B-9 of this Appendix. \ 

The first task element in Figure B-1, Establish Testing Conditions 
for Multiple-Choice Tests (2.2.1.2.2) is the task to be performed. The 
task elements under this block must be performed, or at least considered 
when preparing multiple-choice test it^ras. For example, task element a, 
"set readability level for test," is the first sub-task shown in the 
figure. The asterisk (*) beside the block indicates the availability of 
a non-HumRRO author aid.* In this case, the readability level set is 
contingent oij the reading level of trainees. Since this block is shaded 
it is identified' as a block for. which an author aid was developed in this 
project. This aid is also designated as both an on-line and off-line author 
aid. In the computer version of this author aid, the author Is specifically 
queried as to the reading difficulty desired for the entire test. There- 
after, all material input by the author is automatically checked to deter- 
mine if the desired readability level has been exceeded. 

Block b, "set minimum and maximum number qf answer alternatives 
including the correct answer," indicates that author aids were not devel- 
oped. Block 'c, "determine if more than one answer is' correct," has neither 
an asterisk, nor is it shaded. This indicates that no off-line author 
aid has been identified and no aid was developed in this project for this 



task element. The comment to the right of the block Is Intended to further 
clarify the statement within the block* 

Block d, "set^time limits if any/' is shaded identifying it as requir- 
ing development of an author aid in this projectv In the computer version 
of this aid, authors have three options: (1) an untimed test, (2) time 
limitation for individual test items; and (3) time limitation for entire 
test. Again, the comment to the right of the block is for further clarifi- 
cation. * . 

Block e, "set conditions for test administration," has neither available 
or developed author aids. The comment further clarifies the statement 
within the block. 

This completes the task elements identified as required for establishing 
testing conditions for multiple-choice test items. The next major task to 
be performed is the actual writing of the multiple-choice items (2.2.1.2.3). 
The line. coming out of the block Indicates that in the actual flowcharts 
this task's components are continued on subsequent pages. ^ 

The flowcharts in this appendix are on the pages listed below. 

Block II. 2 (Develop Tests) ~ Page B-4 
Block III. 4 (Develop Instruction) — B-35 
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Figure B-l . Flowchart for Etttblithing Test Conditioni for Multlple-Choioft TMt lt»mt. 



On line 

and' 
Off line 



Ettubllth 'testing conditions for 


multlple-choice tests. 




2.2.1.2.2 






Sit tMKtobillty lev«l for t«ftt« 


< 




•* 


• 






Set mininQum and maximum 


number #f answer alternatives 


including the correct answer. 




*>. • 




— — . — ^ 



Determine if more than ohe 
answer is correct. 



Contingint on reading level of trainees. / 
See: Peter Klncaid, Automated Readfbllltv Index, Human Factort^ 
Society Bulletin, Vol, XV, Mo. 5, May 1.972, 

Author Aid for reducing readability level 



That is, is one (or more) answer alternatives correct, in addition 
to correct answer? 



c 



Set time limits if any. 



Time limit for each item and entire test. 



' - ■■■ ■ ^ ■ ^ ~ " ' 

Set conditions for test 


Will all trainees take the same test? Will they all be in the same room? 


administration. 




Will they be tested in shifts? {Some of these decisions are contingent 






on available facilities and testing personnel.) 




e 










Write multlplf^-chQice items, 

\ * 


Author aid for writing multiple-chdice items. 






Author aid multiple-choice initial preparation worksheet. 




2.2.1.2)a 




1 \ 

*Non-HumRRO author aid. 
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* - 


• 








■ t - ■ • . . ._ . 

ir.2 DEVELOP TESTS 

^' . , , • * *^ 

N" " - 


* 

D^t0rmine how detalM the tmt 
shouM b€. 








- • 






* 

• 


Develop test plan worksheet , 

< 


What informatioh to use for this activity, (See Swezey and Pearlstein, . " 

p. 3 33). 

* ■ f . 




2.1.0.1 












Sample 



Sample objectives at random. 
Keep selection process secret 
to examinees. 

/ 2.1.0.2.1 



Ba^e decisions on resource 
^ availability, time con- 
straints, and criticality/ 
importance of each objec- 
tive. (Guidance on how to 
sample objectives is in 
Swezey and Pearlstein, p. 3-6) 



Translate TLO's and LO's into 
test items. 



TLO defines objective of test,LO's 
are translated into tests (or test items), 



Read over action, condition, apd 
standard fqr each TLO. (Obtain 
from Learning Objective Analysis 
Worksheet.) 

2.2.0.1 



< 



Specify whether high or low item 
fidelity is required. 



2.2.0.2 



Base decisions dn resource availability, time 
constraints, and criticality/importance of 
> each objective. 
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NO 





NO 



Consult examples below to 
choose Ie9rntng category * 



LEARNING CATEGORIES 

Mental Skillt. Skills «uch as: 
Problem solving , 
Concept fornnation 
Decision making 



Mapreading 

Computer programm^tfig 



T^ng lnformatk>n -Knowing st^dard operati^jig procedurts. filling out clerical forms 
Physical Skills— Perceptual motor sloills, such as typing, target shootirrg 
Testing Attitudes— Cooperation, dedication, helpfulness, leadership 
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Tinting Mental Skills 



2.2,1 




On line 
and 
Off line 



On line 



Establish testing conditions for 
multiple-choice tests. 



2.2.1,2.2 



Set readability level for tint. 



Set mininf^um and maximum 
number of answer altematjves 
including the correct answer. 



b 

/ 



Set time limits if any. 



Determine if more thaft one 


answer is correct. 




/ 






c 



Contingent on reading level of trainees. " ^ ^ ^ ' yv ^ * 
S^: Peter (Cincsfld, Autortuited Readj^biJT<V')ndex, HumHin Factors 
SocietyBulletin^Vol. XV,No, 6,May ^97l ' - *v 



Author aid for reducing readability level, i 



That is, is one (or more) answer. alternatives corlrtct, if)' 
addition to correct answer? 



Time limit for each item and entire test. 



On line 
and 
Off line 
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Set conditions for test 




administration. 








e 











Write multiple^choloe items, 



2,2,1.2.3 



T 



*Non-HumRRO author aid. 



Will all trainees take the same test? Wilk theV all be in the same r^pm? 
Will they be tested in shifts? (Some of Oiese decisions are contmg^ryt : 
on available facilities and testing personnel.) • 



Author aid for writing multiple choice items. 

Author aid multiple-choice initial preparation worl^sheet. 
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On line 



2.2.1 2.3 
1 



J — ■ *■ ■ - l> . --.I , , 

^ Mtke sure that Item tem pne > 


obiectlve only. 




V 






a 


I ... ... . , 




. /, — _ 





to oorraqt aniwar^ ^ 



On line 

and 
Off line 



Maka lura that/aadabiilty layal of 
n«m ii not lurpamd^ ^ - 



Check questions for grannmatical 
errors. 



Avoid negatives 


in item stem. 


Avoid using "none of the above" 


as an alternative. 


- 


e 



Make sure thai distractors 
(options other than corredt answer) 
are plausible. 



f 



If only one alternative is correct, 
make sure that it is unequivocally 
Correct-* 



9. 



~ 1 

Non-HumRl^O author aid. 



Use on-line or off-line^ reiidability aid. Off-line formula i$ at end 
of flowchart. / " , ' 

Author aid on reducihjQ'readability level. 
Peter Kincaid, Automated Fleadability Index, Human Factors ' 
Society Bulletin ^ Vol. XV/No. May 1972. 



7 

Errors or phrasing of question might inadvertently indicate 
poj-rwct answer. ' 



Negatives are cohf using to test takers. State items positively. 



Do not make distractors ovefly technical. 
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2,Z1.2.3 



Sllfqt pQ«Jtlon of ooHrrM^^^i^ 
ilttrKiitivi, 



Consult subject matter experts 
or peers for reyiew of items. 



Chosen at random at author's option. ' ' 

Author aid question scoring form for recording corroct answer 
position and number of alternatives. 



They will (^heck item fidelity and correctness. 



r— 1 ' 

Consult test )»xperts for final 
review of test. 



They will point out possible poor constructioin pf items (and 
weak or apribiguous items). ^ 



1 



On line 

and 
Off line 



On line 



On line 
and 
^ Off line 



Establish testing conditions 
for trua/false test* 



2.2.1.2.4 



Stt r«»d»bilitv Itvti tor tf it. 



Determine number or percent 
of items to be true (and 
false). 



Write true/falM items. 



^^1.2.5 



Make sure item tests one 
objective only. ^ 

a 
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^Non-HumRRO author aid. 



Sat time limits, if any, 




c 






Set conditions for test 


administration. 






d 







Contingent on reading level of trainees. 

See: Peter Kincaid, Automated Readability Index, Human Factors 
Society Bulletin , Vol. XV, fto. 5, May 1972. 

Author aid for reducing readability level. 



Will the items be 50% tru^ and 50% false or other proportions? 



Time limit for eacl\ itSm or entire t^st. 



Will all trainees take the same test? Will they all be in the same room? 
Will they be tested in shifts? (Some'of these decisions are contingent 
on available facilities and testing personnel.) 



Author aid for writing true/false items. 

Author aid true/false initial preparation worksheet. 
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Online 
and 
.0 Offline 



2.2.1.2.6 
I 



Mikt turt rMdtbillty itv«l it not 
{HirpMMd In lt*m stem. 



/ 



Check item for grammatical/ 


errors. 


/' 




c 



Paraphrase material for test items; 
do not lift rnateriai straight from 
text. 



^ See: Peter Klncald/Automated Re^ability Index, Human Fectort 
/' Society Bulletlio /ol. XV, No. B/May 1972. 

Author aid forfreduclng readability level. 

% 



See Stevens and O'Neil (November 1974) for guidance 
and examples. ' 



ERIC 



Avoid ambiguous and indefinite 
terms (such as spmetimes). Also 
avoid use of negatives and 
negatively worded stems. 

e 











Keep true and false statement^ 


equQl in length. 






f 




Be sure that item can be 


categorized unequivocally true 


or false. 






9 



Consult subject-matter experts 
or peers for review of test. 



J^Non-HumRRO authior aid. 



They will check test for fidelity and correctness. 

1 ■ 
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2.2,1.2.5 



Coniult teit expert for final 
review of test. 



They will point out poulble bad conttructioh of Items (and 
look for weak or ambiguoui items). 




Off line 



Establish testing conditions for 


matching tests. 






2.2.1.2.5 






Set readability level for test. 




a 








A 


Set number of elements for 


each column. 






b 



Contingent on reading level of trainees. 
Author aid for reducing readability level. 

See: Peter Kincaid, Automated Readability Index, Human Factors 
Society Bulletin, Vol. XV, No. 5, May 1972. 



* Non-HumRRO author aid. 
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t 



2.2.1.2.6 
l_ 



Oetormind if noore than one 
oletiMnt it to Im paired correctly 
with other column element(t). 



ERIC 



Write matching test items. 



2.2.1.2.7 



Off line 



Make sure readability level is 
not surpassed. 



* 








Set time limits, if any. 


• 




• 

d 










Set conditions for test 
administration. 






e 



Make elements in each column 
similar in length and type an<| 
as short as possible. 



1 

*Non-HumRRO author aid. 



Will all trainees take test in same room? Will all trainees take same 
test? Some of these decisions are contingent on available facilities 
and testing personnel. 



Make sure item tests one 


objective only. 






a 



Author aid for reducing readability level. • 

See: Peter Kincaid, Automated Readability Index, Human Factors 
Society Bulletin, Vol. XV, No. 5, May 1972. 



Column items should be of similar category .(e*)|., nouns, verbs). 



4ir 



2.2.1.2.7 

L_ 

Uibel each column. 



Make columns with unequal 
numbers of elements. 



Consult subject-matter experts 
or peers for review of tests. 



This is so answers cannot be found by elimination. 



They will check test for accuracy and f idelity^ 



Consult test expert for final 
review of test. 




Ambiguous items or those of poor construction will be found 
in this review. 
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On line 

and 
Off line 



r 



On line 

and 
Off line 



Eftabllth tettJnQ conditions for 
Constructed Response Tests using 

completion list Items. 

2.2.1.2.8 



Set reSwMiMlitv level for tf st. 



2 



Determine ^Mt>ether nrH)re then one 
eniwer li corrMt end whtttxkr 
•mwert can be pertietly obrrect. 
Determine If there ere entioipeted 
wrbh0 entwert. . 



Author aid for reducing readability level. 

See: Peter Kincaid. Automated Readability Index, . Human Factors ^' 
Society Bulletin, Vol. XV. No. 5, May 1972. 



Author aid for preparation of constructed response items. 
Author aid of examples o^ answer alternatives. 



ERIC 



On line 

and 
Off line 



On line 



*Non-HumRRO author aid. 



Establish fornrMit for answering 


queitloni. 


c 






Establish level of hints to bo 


givenlrainee. 


d 






Set time limits, if any. 




• 






Set conditions for test 


administration. 






f 



Author aid constructed response initial preparation worksheet, 



If listing, number of items required? What pari of speech 
is niissing? 



Time limits fpr^ach item or entire test. 



Will all trainees 'takQ test in same room? At same time? 
Will they take same tQst? Some of these decisions are 
contingent on facilities and available testing personnel. 



On line 

and 
Off line 



2.2.1.2.8 

I 



\M!rltt «amptft(pn/ll«t itiim for 



2.11.2.0 



Author aid for preparation of conitructed reiponie items. 



Make sure item tests qne 

^ objective only. 



On line 

and 
Off line 



Make sure rMdability level of 
Item It not aurpetied. 



Check items for grammatical 


errors. 






c ' 



M a numerical response is required, 
specify units to be used. 



In completiorl, omit only 
key words. 



Author aid for reducing readability level. 

See: Peter Kincaid, Automated Readability Index, Human Factors 
Society Bulletin^ Vol. XV, No. 5, May 1972. 



If fill-in-blank format used, is question understandable 
with blank? 



On line 

and 
Off line 



Lilt posiibi* correct univvtrs. 
U<t •nticiptt«d Incorrect 
imiwert. 



f 



— 1 

^Non HumRRO author aid 



Author aid for preparation of constructed response items. 
^ Authoj^^eW of examples of answer alternatives. 
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2.2.1.2,9 
1 



Consult $ub)ect*matter export or 


peers for review of test. 




g 






Consult t^t expert for final 


/ review of test. 






h 


r 





' They will look for fidelity and accuracy and may add other 
^ correct answer possibilities. ^ 



This review will pinpoint poorly (ibnstructed items or 
ambiguous items. 



2,2.1.5 



^stabliih Testing Conditions for 
ConsfiTIcted Response Tests 
Using Essay Items. 

2,2.1.2.10 



Set readability level for test. 



Author aid for reducing readability level. 

See: Peter Kincaid^Automated Readability Index, Human Factor s 
Society Bulletin, Vol. XV, No. 5. May 1972. 



XDetermine if all items are to be 
u^d or if a choice for answering 
is provided, 

\ b 



Specify number of al^rnatives to be answered. 



^Non HumRRO author aid. 



B-17 



2.2.1.2.10 

I 



Determine If more than one aniwer 
can be correct and If arrtWers can 

be partially correct. 



Set limit for number of words 
to be written for each Iterri, 
If any. 



Determine if test answer will be 

S' 

scored fbr grammar, spelling an4^ 
punctuation. 



Set time limits, if any. 



Set conditions for test 


administration. 






g 



Write E^y Type Items for 
Constructed Response T^sts. 



/ 



2.2.1.2.11 



Make sure iterp tests pne objec- 
' tive only. ^ 



\ 



Will all trainees take tfie same test? In the same room? In 
shifts? some of these decisions will depend on available 
facilities and testing personnel. 



On line 

and 
Off line 



2.2.1.2.11 

__l 



M r- 



Makt suri rtadiblMty Itvti of itt m 
it f^t iMrpifitd. 



Be sure item is phrased clearly. 
Start itenn with action verb, such 
as "Explain." 



Prepare sample correct answer and 
acceptable alternatives, if any. 



Consult subject-matter experts 
or peers for review of test. 




Consult test expert for final 


review of test. 










f 



Authoi aid for reducing readability level. 

See: Piter Kincald, Automated Readability Index, Human Factort. 
Society Bulletin. Vol. XV, No. 5> M^V 1972. ' •'Z 



J 



Observe time it takes to construct answer. 



They check test for fidelity and accuracy. 



This check is for poorly constructed item? and ambiguous items. 



ERIC 



• 4 



Non-HumRRO author aid'. 

• • • • ■ 



/ • 
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1 ' I * '■ 
"A 






1 ■ 

■ ,J 


Establiih Testing Conditions for 


'Oral Tests. 






. a 







Set conditions for test 


adnr^inistration. 






a1 



'O^t^rmine if more thun one 
answer ii correct tnd if tfiere 
will be pirtiftlly correct enswert. 

•2 



Will alt trainees take identical tests? How wiH facilities 
be used? - . r. * 



Author aid of examples of Answer alternatives. 



determine. wtiether administra- 
tor's questions are written or 
oral. 





\ 


Determine whether correct 


answer is a phrase, word, or\ 


exposition! 




^ ■■1.'" *' 

■■■ . \* 
it . ' 1 . — , 


a4 " • ^ 


■V 





Will verbal ability be part of answer evaluation? 



Establish level of>hinU to be piveri' 




Are\)umber of components in answer to be delineated? (For 



' e^mi 

A ''a 



xplain the three parts of an expeViment, etc.) 



2.2.1.3 



Online 
and 
line 



Choose item format. 



b1 



Make sure item tests one 
objecth^ only. 

^ b2 



Author aids for^praparation of multipla-chpica, true/falia and 
constructed ratponta Itams. 



Review 2.2.1 .2.3, .6, J, .0, and .1 1 to choose bast format. 



r •) 



ERJC 





Makf lura raadabilityjaval ii 


a 

f 


On line 


not iur|^a$std. 






and 








Off line 




b3 




• 






1 




Answer items «to measure 


Check with time allotted for test. 




time required to answer. 


• 










« 










List all corract insert. 


Author aid of examples of answer alternatives. 


On line 














■ 4 












ft 

Consult subject-matter experts 


They check for fidelity and accuracy and may add other correct 




or peers for review of test, 


ansyuer possibilities. 




% 


b6 


' } : 




A 






*Non HumRRO author aid. 













,1 ., X 



2.2.1.3 



Coniult tfiki UKRfrt for final 
Ni^vlew of test. 



b7 





Thil will highlight amblgudui and poorly conitrMPt«d Itamr 



ERIC 



Develop Behavioral Performarxje 
Test. 



2.2.1.4 



Establish testing conditions for 
behavioral performance tests. 



Determine length of each 
exercise. 



a1 



Identify facilities to be 
utilized. 



a? 



/ Find out if any restrictiohs are imposed on use of facilities 
and personnel. 
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•A 

■A 



ERIC 



2.2.1.4 



Identify pei'^onnel to be 
4 Utilized. 



a3 



a4. 



Oe^ermjnejf^pefformance is 
process or product. 



aS 



Establish use and level of hints 
to be provided during testing. 



a6 



FVepare Behavioral Performance 
'^Test Exercises, Descriptions. 



Detefmine sequence of exercise. 



Describe the performance(s) 
required for each objective. 



bl 



Describe steps for correct 
performance ff process ii to be 
measuresd. f 

b2 . 



Set up scheduled use of specific areas and ^h^ule for personnel 
involved in testing. 



"County Fair" type testing environment might be used. 



If process, the behavior will have sub-compK)nents ^ 



Include time restrictions and facilities tQ be used in descriptions: 
Describe locatipnfe and duties of testing personnel In description. 



See'Vin^berg and Jaylor, 1975. 
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2.2.1.4 
i 



Describe attributes of acceptable 
finished product if product is to 
be measured. 

b3 



Perform your own exercises to 
' check out facilities* timing and 
"personnel needed. 

b4 



De^ribe acceptable hints to be ^ 
given if hints are allowed. 



b5 



Develop checklist of behaviors 
to be performed. 



b6 



Consult subject-matter experts 
or peers for review of tests. 



b7 



Consult test expert for final 
review of test. 



b8 




This will be neces^ry for scoring later on. 



They will check for fidelity and accuracy. 



/ 

^ / This check will illurtiinate ambiguous or poorly constructed 
./ exercises^ 



On lino 

and 
Off line 



Writt T9$t Ihitructiont. 



2.2.1.8 



Prepare instructions #or 
administrator(s) of tost. 




Author aid for preparing 
tost Instructions, 



On line 

and 
Off line 



Pr«paf« initial «nd<within test 
instruction^ for trainees 
'taking test. 




Check readatJility level oi^instructionvto guataptee that reading, 
level has ni\X been surpassed. ' " * • 
Author aid jfor reducing r^dability level. . ' • 

Author aid for developing test instructions. - * 




Y 



4 



Testlog Information 



2.2,2 



Determine item format. 




Item formats are the same as those in the Mental Skills section. 
The types of mentatskills tested in this category are recall and 
recognition. The best formats for tet^ing information are multi- 
ple choice and matching (for recogoitipn), and a completion 
type of constructed response (for recall). 



Testing Physical Skills 



See behavioral per formance test development activities. 



I 



Testing (MeMsurlng) Attitudet 



2.2.4 



Determine whether attitudes are 
to be measured by observation 
Of by solicitation from trainees. 



Determine scaling techniques 
to be used. 



a1 




Select measuring technique 
for attitudes to be observed. 



Describe behaviors which . 
demonstrate given attitudes. 

4 

bT 



Establish rating system fdr 
adnriirtistrator(s) to^use.^ 





Check section 

2.2,1.2 
especially comple- 
tion & true/false. 



These measures 
present statements 
which trainees either 
"tgrel" or "disagree" 
with (or "like" or 
''dislike). 



A good source for choosing observation instruments is Mirrors 
for Behavior , edited by Anita Simon and E. Gil Boyer. 



Determine if trainee will'be rpted by one or more raters. 
Create rating sheets for raters to use:* 



* l5Joi}-HumRR6 author aid. 
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Set Training StandardM 



2.3 



Determine if test Is for end of 
training or within training. 



.Determine level of proficiency 
warranted within training. 



I 



Decide whe?>Nir performance 
at end of trairtng should equal 
or surpass the job performance 



measure. 



Establish criteria for trainee 
performance of LO's * 



False Positives and False Negatives. 



2.3,1 



Perform validity check. 



a 



Setting of criteria is sometimes determined by reference 
to job requirements or consensus of '^experts " 



Sometimes trainees are required to "overlearn^' so that decay < 
of learning on the job is not detrimental to performance. 



Decide on number of LO's to be met by trainee. 



Validity shows discrimination^ beJtween masters (those vyho are "c 
and non-masters (those who ^e "no-go"). See forinula at end of 
flowchart. » 



ERLC 
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2.3.1 

J 

Decide on pitfalls of eitfier 
folte positives or false negatives. 



RanJi Order Students (if required). 



2.4 



4r 



List scores from highest to 
4owest . 



Provide identical ranks to trainees 
having Identical scores. 



t). 



Set Cut off Scores 



2.5 - 



Reviev\/ training ^andards a^ 
established (in'2.3) to set 
cut off s(jores. 



Set cut-off scores, recognizing 
probability of false positives and 
false negatives. ^ 



Decide which type of false situation is more critical and 
which can be tolerated. , ^ 



. Develop form for listing purfio^s. 



Only If required. 



Swe^ey says. "If the cost of a false positive (passing an 
incompetent man) is very high, the cut off point should be 
set very high." (p. 613) 



t9 



Lowor cut-off scores tt required 
by manpower needs or critlcality 
of task. 












Determine qualitative scorinn 


procedures. 










a 






Determine whether •partially 


correct responses will be 


ovaluatetl, 






al 










Determine scheme for assessing 


partially correct answers. 




a2 






. Determine whethQr presentation 


of response, grammar, spelling o( 


punctuation will affect response. 


V • 






a3 




1 

* 







Quant. 




Decide what effects the scoring can have on trainee comparisons. 



Especially applicable in constructed response (listing) format. 



Oral tests.and construotetl response-essay. 



90 



B-30 



2.6 



'Outline soaring guidelines for 
behavioral checklitti or scales. 



84 




Checklist scoring is aplplicable to "(process" types of 
performance. ~ ' ' 



Determine quantitative scoring 
procedures. 



2.6.2 



On line 

and * 
Of fr line 



Oh line 
. and 
Offline 



Determine number of points 
for each response. 



Determine if responses ar'e to be 
rank-ordered. 



E^bbfli^ penalties for 
incorrect responses. 




1 




Author aid for scoring test items. 



Could be used for mi^ltiple-choice or constructed response, 
wlwfe^some re$poDses are considered ''more corrett" tfhnj - ^ 
ojthers. VV^ighted responses? 





Especialjy applicable in Ture^alse. or Multiple-Choice where 
probability of .guessing correiHIy^fe^niigh. Could also be used 
in matching format. Formula or correction, for guessing. 
^. Aut}>or aid ^or^ scoring test items. 



ERIC 
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^ 2.6,2 



What wnttlbt <yitmt«d. 



.On line' 

and 
Off Hhe 



EI 



pfWmtniii.l^in* for •«M«iln0 



Outline scoring ftuldelines for 
behavioral checklists and atti- 
tude scales (if used). 



f 



Cmiermine when scoring wil 
occur, aftqr or during test. 



2.6.3 




Determine whether scoring will 
be done by hand or machine. 



V2.6.4 



Write scoring dirffctions. 



2.6.5 



Author iid forscprlng test items. 

Agthbr 9id of examplei of'aniwer alternatives. 



y 

Especially iH^plicable Ir) consfruij|ed response (listing) format. 
Author aid fo^scoring test items. ' > 
Author aid of ex^jimples of ansv^^r alternatives. 

I. 4 4 ^ 



Will interference scoring be M?ed? See Swezey, p. 6-6. 
Go/No Go typo? . * 



Develop for^ for recording cprrect;,and incorrect answers of 
class and individuals. Develop method for determirtation of 
number of correct answers (of a scoring key). 



1^ 



Perform U^:^]p9 procedures 
tryout. P*rf^m reedebility 

2.6.8 



f Non-HumRRO author aid. 



Author aid for reducing readability level. 

See: Peter Kincaid. Automated Readability Index, Human Factors 
Society Bulletin, Vol XV. No. 5, May 1972. 
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Ma 



... * •. 



R«viitt'proc«durrt ai irtdicated 
In tryout. 



2.6.7 



Collhct BMlinf Data, 



2.7 



Decide hbvy data are to be 
stored, hand or machine file. 



Write instructions for storage' 
Of data. 



Write instructions for gathering 
background^ data. ^ 



Collect background and train- 
ing data for each trainee. . 



Usqally npt test d^eloper's responsibility .. 



1 



Develop form for intermediate storage of data. 



' . . . o ■ 



/;\\- 
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. _ Ill;4 PEVELOP INSTRUCTION 

1.0 Jntrbductioh ^ " 
2.0 Proc^ur^s 



— " — " ■■ — 1 , " .. 1 II 

Identify instructional needs 


and constraints. 








... 


• ^1 



trainees. 



Determine number of 



Determine entry characteristics 
of trainees relevant to instruction. 



Erg., reading (Comprehension level. 



Identify budget, restraints for 
instruct ior^al development. 



Identify time constraints for 
instructional development.. 



Z1 



Identify tcheduHng co.nftraints, 



Determine fidelity required during 
instruction. 




See LOAW 



1 . : I 



' > 



9S 
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Identify 9V9il9bl9 mourcBS. 



2.2 



Identify personnel knowledgeable 
in various modes/media of 
instruction. 



Prepare roster of coworkers 
(peers) available for instruction. 



a1 




Prepare roster of availeble, 
^.qualified instructors. 



Describe and document 
available facilities. 




c 



List all library /reference rooms, study rooms, and equipment; 
classroom and large lecture halls; practice rooms/laboratories. 



ar>d equipment'. 



9/ 
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On line 

and 
Off line 



On line 

and 
Off line 



fMv9kH> Inttruction, 



2.3 



Prepare outline oftLO'i and LO's' 
to be covered by initruction. , 



/ 



Determine mode of instruction 
for each section of course. 



A^\fi ^thln count 

|h«rt Will i>6 pr9K?tiot. 



1- ' 




Select first mode of instruction 


for development. 






d1 


1 — 1 




Go to block explaining tnpde 


selected. 




•* 

* 


d2 



Base dacisioni on critiballtv of task, fidelity required, equipment 

availability, personnel pvailability. 

Will reaming be self-paced Oearqer controlled*, tutorial, small- - 
group or large-group discussion, or teacher oriented? 



Author aid for preparing learning objectives. 
Author aid for sequencing of materials.. 



V 



Author aid for preparing practice frames. 



CHOOSE FROM LIST: 
2.3./ Audio script 
2.3 J Video (naterials. 

2.3.3 Audiovisual materials. 

2.3.4 Printed materials. 

2.3.5 Programmed initruction. 

2.3.6 Platform lectuhs. 



r i 



2.3.7 Self t9acbinaitxportabh 
package (Sfeh}. 

2.3.8 Supplementary instruction. 

2.3.9 Adjunct programs. 

2.3. 10 Job performance^ aids (JPAs). 

2.3.11 Formal OJT. 

2.3. 12 Other forms ofrnedlatttd 
instruction. 

2.3.13 CM. ■ 
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miifng qp tiudio $cr/pt okff] 



2.3.1 



Idontif )f lectioni of CQiirse for use 
of audio^ln conjunction with other 

a 1^ 



✓ 



Identify comprehension level 
of trainees. 



"A 



Using outline, prepare plan 
for script. 



1 



Further outline TLO's and LO'^ 
pertaining to this Section. 



c1 



Plan use of audio cues, music/ 
voices and combinations of these. 



c2 - 



Set up schedule for script 
development. 



*Non-HumRRO author aid. 




Bvliw TLO's and LO's. 



See brown, J.I., and Carlseft, G.R. Brown Carlsen bistenfnp 
Comprehertsion Tert. New York: Harl^urt, Brace and 
World, 1955. 
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Establish ti^e fra^ne for 

development. \ 



d1 



.Me^^-wkh^audio equipmertt 
personnel to set up Recording 
schedule. 

d2 



Write script from outlines. 



' Use two sided script form f oiu 
writing cbntent. ^ ^ 

I el 

I 



Indicate pauses and uses of 
other media on script form. 



e2 



Have script reviewed. 



. Give script to pser for review. 



f1 



ERIC 



Left side should contain special in^lructions for cues to music; other 
sounds; right side contains actual scfipt and directions to narrator. 
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loo. 



2.9.1 

M««t wJth produce to raview 
tcript. 



f2 



irt: 



Revise icript if necessary to 
^ jchange pace, clarify sections, etc. 



P^epere required|^mbef of 
copies of script for productior* 
purposes. 



Wake recor^iing of script. 




Pr9p§rlno vtd0O<mfy nmwriil$. 

2.3.2 

I ' 



Identify sections of course for 
use of video in ppniu^fion with 
the|[ media. 



♦Review TLO's and LO's. 



Prti|>are outline of instructional 
sections applU:able to video 
medium. 



Sketch or describe visuals 

to be presented on storylwards. 

■ - ■ bi 



Specify details of each 
picture/illustration. 

b2 



Include^pecial effects, lettering, shading, amount of detail. 



Ensure that detail Is correct and 
not too complex or cluttered. 



b3 



Establish sequ&nce^of visuals to 
be presejited. > 

c 

' ' I ' > ' 



r 



ERIC 



1 



V 



/ 4 



/ 



2.3.2 

« 

Identify timing of tequenoM. 



Identify locatloni wtter e video^ 
only materlalf are to fit with 
othei- Instruction, 



.P2 



?et up schedule foV developmen 
»of materials. 



Establish time'^frame. 



til 



_Mee^with illustrator /photograpKer 
to e)$plain storyboards and- • 
sequence of visuals. % 

d2 



Arrange for TV production 
facility to produce materials. 



d3 



Determine what TV^ilities are required and/or available, 
(i.e., studio equipment. personLnel. materials, etc..) 



ERIC 



Have video materials reviewed. 
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2.3.2 



Giv* materials to pear for review. 




•1 


* ^ 








Meet witb pi^oducer to review 


materials, ' 






•2 



' Revise materials if necessary to 
clarify sections, change pace, etc. 



Have requireit number of copies 
produced. 




r 



Tff HK/ioviiM production. . 



< 



2.3.3 



The 9udio in $udlovltU9l. 



2.3.3,1 



Identify sections of course for 
^diovisuat media. ' ' 



Identify comprehension Idvel 
of trainees! 




Set up schedul^^for script 
development/^ 



Establish tifne frame. 



c1 



Meet With audio equipment 
personnel to set up recoi^ding 
schedule. ' 

s 

c2 • 



T 



*Non HumRRO author aid^ 



Review YlO's and LO's. 



7 



See: Brown, J.I., and Carlsen, G.R. Brown-Carlsen Listening 
Comprehension Test. New York: Harcburt,»firace and 
World, 195&. • ' 
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2.3.3.1 

L_ L 

Prepare outline of Inttructional 
lactiont applicable to audio- 
visual medium. 



1?^ 



Write script frort^ outline^. 



pse two-sided script.form 
for writing content. 



el 



I 



Indicate pauses and uses of other 
media on script form. 



e2 



Have script reviewed. 



Give script to peer for review. 



f1 



Meet witii producer to review , 
script. 



f2 



Left side should contain production instructions (use of music, 
special cues); right side should contain directibns to narrator. 
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V 



2.3.3.1 
_l 



R«vl«e icf rpt If rMcMs^ry to (iNinQs 
pace, cltrlfy sactiont^ etc. 

' 7 . . 



Prepare required number x>f 
copies of/ script for product^pn 
purposes. 



Record audio materials. 




Seg Closed-Circuit Television Production Tochniques . by L.G. Goodwin & 
T. Koehring. Hndiana^olls: Howard W. Sams & Co., 19r0). 



ERIC 



The visual in audiovisual. 



2.3.3.2 



I 



Prepare outline of instructional 
sections applicable to visual 
medium. 



r 



Sketch or describe visuals 

to be presented on storyboards. 

a1 ' 



T 



*Non-HumRRO author aid. 



f^eview TLO's and LO's. 



B-47 



t I 



ERIC 



23.3,2 
I 



Suloif y datalU of aaqh 
pie^ur'iB/illuitration. 



a2 



Include special effectt, IttterInQ, •hwllno, amount of detail 



Ensure that detail is correct . 
and not too conjplex or 
cluttered. 


- 

- 




a3 • 










Establish sequence of visuals 
to be presentfd. . ' - ^ 






b* 










Identify timing of sequences. 




% 


b1 





With first versjon cff visuals, 0t . 
play audio accompahiment vrtiile 
going through sequence. 

^b2 



S^t up schedule f|pr 
develppment of materials. 

I ^2 



Establish time frame. 



b3-1 



'..m"'.:^ 



i 

V 



108\ 



) 



2.3.3.2 
"I 



Meet wrth illuttratqf /photographer 
to eirpl^in itoryboardf and 
9oquance ot vitualt. 

b3-2 



Arrange for phgtographic 
produ6tion o'f yisuals. 



b4 



! Determitle what |:^o^raphip tocilitiet required and/or 
I ^vai|et4e |l,e., studio, equipment, personnel/ materjali, e}c.^ 



' Have'visuals reviewed. 



b5 



Give materials to peer for review. 



b5-1 



Revise rnaterials if necessary 

to clarjfy s^tions, change pace, etc. 

b6 



Have required number of. 
copies produced. 



b7 




See The Video HandbooM VlY: Media Horizons, 1972. 



*Non-HumRRO author aid. 
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/Vo^/ii^]^ # flMe/t9p§ program. 



r 



2.3.3.3 



Determine eppropriate timing 
for each storyboarcHn outline. 



a1 



Integrate audio tape recording 
with visuals. 



Refer to audiovisual outlines. 



ERIC 



Record pauses, audio cues, 
voices, and combinations of both. 



a2 



Hav^ illustrator /'photographer 
produce^slide 




-n — 

Record audip^portion on^tajie 



m^uding 



cues. 



TV 



Revifew inwated slicje/tape 
progranh./' 



7*~r 



^*NQh-HMfli|hRO author aid. 



See The Video Handbook, New York: M^ia Horizons, 1972 
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2.3.3,3 ' 
I 



Present program to peerf.for ( 
review. 



b1 



Revise audio and/or visual 
materials if necessary to clarify 
sections, changes, pace, etc. * 



Have required numbers of slides 
and tapes' prpduced . 





Not recommended. 




•Follow ttepi for 2.3.2 (Pr«p«HnQ vWeo matwlali). 
Saai Clofd-Clrcult T«ly^Ulon Production Technique . 
Indltrtapollt: Howard IV. S«m« tpi6 Co., 1970. 



1 



Developing printed materials. 



2,3.4 



Prepare outline of instructional 
sections applicable to print medium. 



Determine sequence of 
instruction. 



m 
s 



Review TLO's and LO's. 



Set time limits for using 
materials in class. 



b1 



*N<y HumRRO author aid^ 



112 
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^3.4 



1 



0*t«rmrne location and type of 
practical exarciwt and illut- 
trationti 



b2 



* ■ 




Er|ci 



1 1 

I 



I Identify readk^comprehen^ion 



ldv«J of trainees. 



On Jine 
and 
Off line 



Schedule preparation of 
materials. 



1 



Set time frame for preparation 
and production. 



el 



Identify available facilities. ^ ^ 


\ 






.e2 



Identify qualified personnel to 
develop instruction. 



e3 



^^fbn-HumRRO author aid. 



Use Reading test scopes. 



Author aiti for reducinfj readability level. 

See: Peter KIncaid. AutOrrwted Readability^ Index, Human Factori 
Society B^illetln/ Vol. XV, No. 5, May 1972. 



i 



1 < 
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^13 



ERIC 



2.3.4 

i 



I 


Meet wllh edltor)al end production 
tt^ff to ditcutt 0lant for Inttruc^ 
tional development. 






•e4 






> 




. \ . 1 : _ 

Write In&tructlon. ^ 


Cad- RrkhArt l!^Mnninn Hnw tfk Tllkfl thl) FoQ Out of WrltinO. 

Chicafo: The Dartnell ^orp.. 1964. 

. F - . • ■ ■■ . 


> 




• . V - "■ ■. • 










WritA introduction and 
surTimary of materials Tor 
trainees. 


See W. James Ppph^jn and ^va Baker, P<anninoan Instructional 
Sequence, EnglewiioA Cllff?^ Prentice Hall, Intf^, 1970. 


\ 


f1 





Indicate outside references 
and any other aids used. 



f2 ^ 



\ 



Prepareftremedial versions of 
instruction if needed. 



f3 & 



Write primary instructional 
content. 



f4 



Tyler G! Hicks, SuccessfurTechnlcal Writing, New York: 



McGraw-Hill Book. Co., 1959. 



Write practice exercises 
and self-tests. 

t 




<. 


f5 




<* 


1 

*Non-HumRRO author aid. 


4 
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. -2.3.4 , 

L 



1 "TT— 

Have printed material revtewecf. ' 



Give te peer for review. 



Revise if necessary to clarify 


sections, etc. 




• 

N 1 






h 



Have sufficient copies prepared 
for use by trainees. 




2.4 



7 1 




D0¥9lopfng program tmd 




instructfon. 








" ' . Z3.5 










.■■v..,4.v..M., 










^.^:.]^^ 






\ 


.i ■ 








On lir\0 






■; V.-. 




and 










Off line 


■ \ 

w. 









^ Organize frames into logical 
order. 

a1-1 



Plan franie size contingent on 
type of trainees. 



a2 



rteviaw TLO ttandardt and cpndltioni.. Review manaaement 
plan. See: Thlaparajan The Proflrammlng Proce»»; A 
VracticarGulde, Worthln(|ton, Ohlcx Chaflei A. Jone» ^ 
' Publiihlng Co., 1971. 



Author aid for preparing learning 
Author aid for seq|ienclng of materials. 



obje^fvet, 



Frames on the main track (if there is individualization). ihMid 
present all' the information that a student needs in order to 
.master the subject matter. 



F^elate frame size to^expiacted student behavior. Determinahow 
large a step toward mastery the student can reasonably be 
expected to take in each frame. See: M^rkle, Ggod Frames 
and Bad, New York: John Wi|^ & Sons, 1964. 



Offline 



ERLC 



Matcf a rough draft of frtme 
indUdlna Mluttrationi. 



.•3 



Draw flowchart of frame 
sequence!. > 



a4 



T 



*Non-HumRRO author aid. 



Author aid for preparing practice quelstions, Author aid for 
instructional frame development. 



Us 
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2.3.6 
I 



Lopate areas for practice axarciiei^ 



a4 1 



On line 

and' 
Off line 



v.-;^^: 



;5 O 

ERIC 



Determine the amount of practi6e necemry beyond the 
minimal range of examples. See: Markle, 1064. 



Multiple choice/ true/false, completion? See: MarkJe, 
Good Framei and Bad , New York:! John WHey & Sons, 
1964. 

Author aids for preparing multiple choice, true/false^ 
and constructed respon^ pry tice questions. 



Locals potentially difficult 
frames. - .f 



a4-3 



J- 



l 



Locate areas for review, rest, and 
self-test. List referents and demon 
stration'materials to be used by 
student. 

a4-4 



Where the program is exj^nsive enough, provide isolated review 
and test items as feedbjack to the programmer, as well as the student 
on how well the teaching sepuence has gone. Prepare quiz which 
tests students umsjerstanding of the mc(}erial. 





* Determine type of prompt to be 
used in each frame. 


Make use of the thematic prompts (in context). Only use formal 
V" prompts (multiple-choice format excluded) when absolutely 
necessary. See Thiagarajan & Markle. 






a5 






\ 








Determine if there will be 
branching. 

4 


If the student is branched to remedial instruction, the material. ^ 
K should represent a restatepnent of information covered in ^ 
the main track. • 






a6 














Set sequence for brand;) ing. 
* • 


0 

• It 






a6 1 


<. > 

I 

• 



T 



Non-HumRRO author aid. 
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il7 ■ 



Determine If there ere supple- 
meYitary materieli. . » « 



a7 



Determine If there are yuppie- 
mehtarv cfxercises. 



a8 



As 



Identify the types of responses 
to be Wtcl^ed in each Jrame. 



a9 



Determine feedback to be given 
to trainee after responses. 



alO 



Write frames in rough draft. 



Provide initrUctfbnt on their use. 



To be used with supplementary materials? Alone? 



Will feedback contain explanatory material? Will examples 
be ysed? v 



See Nesbit and O'heil, Thiagarajan & Markle, 



Write frames clearly in good English, 



b1 



^ Avoid introducing morfe points than can be. responded to in 
any one franje. See Markle. 



Provide good examples on the 
instruction. 



b2 



1 

*Non HumRRO author aid. 



* Provide examples covering the variety of conditions the student 
will cope with. 
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On line 
and ' 
Off line 



11' ,' ■ 


J ■ V - 1 


Check franiei for accuracy. f 


* 

• 


b3 




















^'. vs. ■ ■ .-.■•v ■; 










Eliminate irrelevant material 


from frames. 






'b5 ' 






Writi pr««t)0i tx«r(|lt«k iii 














Choose exercises that are relevant 


to instructional content. 




Cl 






Choose the number of exj^rcises . 


for the frames. 






c2 



,See: Peter -KincS^d/Xutonruited-ReadabUltv Index, Human 
Factors Society Bulletin, Vol. XV, No, 5, May }9727^ 

Author aid for. reducing readability level. ' ' 



See Marl^le. 



Do not make exercises too long at one time. 



Write exercises clearly, in good 
English. . 



c3 



I 

' Non-HumRRO author aid. 



119 



2.3.6 

I 

Have pMf^ review rough draft 
of proflram. 



^.^y I " ~ 

•Tjv out draft on other 
authors. 



d1 



Have editorial review of 
instructional materials. 



d2 



Revise materials as necessary. 




See Thiagarajan. 



See Nesbit and O'Neil, Thiagaraian and Markle. 

I. 



V 



*Non HumRRO author aid: ^ 



120 
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V) 



Dwibping platform Itctuht. 



I' 



ERIC 



2.3.6. 



tdne of instructional 
fectlon\ to be covered* by lecture. 



Divide outline into lecture 

periods of minutes each. 



a1 



Determine time and placefnent 
of tests, quizzes, discussion 
periods, practice, other mediated 
instructior]^ etc. 

a2 



Identify sequence of LO's to be 
covered by lecture. 



a3 ' 



Prepare outline and exercises for 
alternate lectures if they are to 
be devQlopey for different student 
populations. 

, a4 ^ 



Prepare lecture notes to aid 
delivery. 



a5 



Review TLO't and L0'$. 



Refer to management plan for time constraints. 



Refer to management plan. 



Note different concepts to get across. Also, note where 
supplementary materials are useful. 



12 
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a.3.6 

, , i: 

I . ■ 

. Detwmlno d«gr«e of student 
pdrticlpation in lecturoi. 



b , 



SchttdUle facilities and 
resources. ^ - 



Determine c|a$sroom/l^ture hall 
requirements and schedule their use 



c1 



Will students listen only or will Interactions be encouraoed? 



1 1 



Identify supplementary materials, > 
If any, to be used in conjunction 
with lectures and order them. 




Identify instructors/lecturers 
who will present instruction and 
prepare roster. 

c3 



Practice lecture in "dry run" to 
determine how much time ft takes. 



I>repare additional notes and/or 
modify outline^ if necessary. 



d1 



122: 
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J 



Try put lecture 


on other 


ijuthors. • . 








e 






Revise lecture outline, notes. 


exerciteS) etc.. as necessary. 




f 








On line 
and 
Off line 



'develop self -teaching exportable 
packages (STEPs). 



2.3.7 



Develop outline of instructional 
sections to be covered by STEPs. 



J^ntlfy !i»flKieru5elDif tO'i to be 



Review TLO's and LO's. 



Author eid for preparing learning objectives. 
Author aid for sequencing of materials. ' 



Select places for exercltet and 
«a)f-tetti. 



b1 



Identify the supplementary 
^teriajs and references to be 
uied with instruction. 



Write introduction to include 
list afhd sequence of LO's to be 



attaihe^. 



d1 







Write draft of STEP. 




d 









See: Detarline Assoolates. How to Design and Develop Self 
and Supervised InstrjJCtion: A Guide ^r Developing 
Correspondence Instruction, February 1975. 



See: Preparing Extehsion Training, TR ADOC Pamphlet 350-31 , 
(Draft), February 1976. 



Establish standards of performance 
neede(yo complete instruction. 



<I2 



List references supplied and 
describe their use.s 



d3 



V: 



Assembly remedial^and supple- 
mentary instructional materials. ^ 



T " 

\*Nop^HumRRO author aid. 



^24 



2.3.7 

^ I 

Include •xtrcltei and Mlf'l%itt 

with tcorlng inttructiont. 



d5 



Supply branching directives con^ 
tingent upon retpontet to 
exercises and self-tests. 

d6 



Have STEP tried out by students 
representative of target population. 



Revise STEP if necessary. 




Provide discussions regarding all responses. 
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^25 



imtruoiion. 



2.3.8 



Survey available rttourcea for 
validated initructional materlali 
relevant to LO't to be covered. 



a 



Prepare outline of inttruction 
to identify placet for use of 
supplementary instruction. 

. b 



Relate self-tests and exercises 
to supplementary instruction. 



Modify supplementary instruction 
to be in a form similar to that in 
existing program. 




Review LO's for requirements. 



Identify tpeclal Instructions for their use, if any. 



ERIC 



i26 
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y 



1. 



ERIC 



\ 



On line 
and 
Off line 



D9V9lof>lng m^funct Initrvctii 



2.3.9 



Develop outline of initruction 
to be covered by adjunct 
'Initruction, 



Identify sequence of LO't to- 
be covered. 



Identify locations In sequence 
for self-tests and exercises. i 



b1 



Collect and review all materials 
to be used. 



Miktiuri ttm 1)Mt 



Ensure uniformity of length and 
difficulty of Instructional unit. 



c2 



n 

•Non HumRRO author aid. 



V 



Reuiew TLO^s and LO's. 



See: Peter Kincaid, Autompted Readability Index, Human 
Factors Society Bulletin, Vol, XV, No. 5, May 1972. 

Author aid for reducing readability level. 



13.9 

_L_ 



Writf 'procedural Inttructlont 
for trainee. 



d 

I : 

LIU TLO'i and LO's to be atlalned 
' by trainee. 



T 




Write correct reiponsei for 
student to compare his answers 
on quiz or test Items. 

d2 



Prepare explanations for 
possible responses. 



d3 



Prepare branching or remedial 
instructions based on responses. 



d4 





Provide suggestions for remedial 


exercises. 




n » 


d5 






^ / 




Review instruction. 

\ 




e 



I . 

Hivt pMT rfvitw mattrlalfl 

for cccuracy and ippropriattntts. 



el 



Have fltudentfl representative 
of target population try out 
materialfl. 

e2 



Revise instruction, if necessary. 




\ 



Job Performance Aids (JPA's). 



2.3 10 



1 

Prepare outlin^of instruction . 
whicinttentif ies small steps 
requiring one specific action. 



Review TLO's and LO's 




Note reference pnd supplementary material relevant to each unit. 



129 
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L 



2.3.10 
I 



Dfvelop JPA't for unit. 



On line 
and 
Off line 



Plan JPA content Including 


checklists* 


Cl 




Plan use of visuals/illustrations. 


In JPA't, 


/ 




c2 . 






Meet with production personnel 


to discuss layout and content of 


JPA. • 


c3 






' Schedule production of JPA. 




> 

c4 






Writ* IfMrtructl^i for JPA uwr. 

\ 


m 






Review JPA. 


d 



r 

* Non-HumRR0 author aid. 



See: Fully Procedurallztd Job Performanct Aids, Handbook for JPA 
Develogert bV Joyce, etjK, Air Force Human Retouroat 

Lab,AFHRL-TR"73-43(lll). • ' • 



Include references.' 

Author aid for preparing test instructions. 



B-70 



— 

Havt pMT rtyiew JPA by perform- 
Ing taricf uiing JPA. j 

d1 



Have iWivlce perform tasks, noting 
any problems, ambiguities. 



d2 



Revise JPA, if necessary. 




V 



Developing formal OJT (fOJT). 



2.3.11 



Develop outline of instructional 
sections to be taught by FOJT. 



Review TLO's and LO 



2.3.11 

- I — 

Identify which ttotlons will be 
prmnttd •• demonttratlont, 
iMturai, "h«ndi-on" performtncf , 
or nwdletsd Instruction* 
b 



Establith wquence of 
instruction. 

c 


. 




Choose location 
and tests. 

T 


1 for practice 
cl 



Select reference materials 
and aids. 



; 



c2 



Prepare FQJT content. 



■3 



ERIC 



Obtain assistance from subject 


matter expert. 






d1 


t 




Prepare introductory 


material. 






d2 



J 



2.3.11 

6 



1 



Vtfiite procMlurtf for demon- 
ftratlfni and performance 
portioni of inttruction. 



d3 



Prepare practice exercises 
and tests. 



d4 



Prepare lectures and audiovisuj^l 
portions of FOJT. 



d6 



Establish time limits for 
tasks and demonstrations. 



See blo^Kv 2.3,3' and 2.3.6 for further reference. 



\ 



Establish standards for perform- 
ance throughout the training 
period. 



Prepare performance checklist 
for use of supervisor to evaluate 
student. 

fl 



Prepare instructions for 
supervisor to score student 
performance. 

f2 



ERLC 
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On lint 
Offline 




[ 



Review FOJT. 



llave kht>w(edoeable perionnel 
review instruction. 



hi . 





■ ^ 


Try out FOJT in field site using 


trainees typical of target 


populations. 






h2 




Author aid for preparing test Initruotlont. 



134 
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Oth0r forrm^ of rmdiMd 
ln$trvct/on. 



2.3.12 



Devaldp outline 0|f lectiont to be 
dbyaVea^^V otheriormt of medl- 
atfd intti(«i|^on (e.g., CAI). 



Identify available resources and 
facilities to use in instructional 
development. 



Identify available personnel who 
are knowledgeable in using 
resources tQ prepare mediated 
instruction. 



Review TLO's and LO's. 



Prepare instructioH'al sequence and 
strategies depending on charac- 
teristics of mediated instruction. 



Establish branching, sequencing, 
and remedial paths for traPnees. 



dl 



Prepare text, audiovisual illus- 
trations, exercises, tests, etc. 
that are suitable for the mediuro 
to be used. 



J 35 



On lipa 
and 

Off line 




Revjew instruction. 








; f 





Ha,ve peer review materials 

for accuracy and appropriateness. 



ft 





i ~ 


Have students representative 


of target population try out 


. materials. 




« 


^2 - ■ 




See: Peter KIncaid, Automated Readability Index, Hu man 
Factors Society BuHettn, Vol. XV, No. 6, May 1972. 

Author aid for reducing readability levtl, ; ^ 



136 . 



ERIC 



*Non-HumRRO author aid. 
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f/rtt drwft mBtmi§l$. 



2,4 



Choose a complete sequence for 
testing (about 1/2 hours). 



Select sample population similar 
to target popu I ation . / 



Saa: Harwibook for Datlgnars of Instructional Svitams , 
Air Forda pamphlet AFf'so 58, Volunoe IV, July 1973. 



Choose naive subjects. 



b1 



ERIC 



Ihitrtict Si to nota probtami 
in undif (taodin^ ii^ruetion, 
illUitritimii. 



Time Ss as they progress 
through materials. 



b3 



Administer test of materials 
to trainees. 



— ^ — 

*Non-HumRRO author aid. 



Author aid on attitudinal questionnaires for CAI. 



\ 



See: Handbook of Procedures for the Design of Instruction , 
Leslie J «Brigg9, Pittsburgh: Am, 1970. 
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V 



Ch9ok to m if inttruotion 
ii top long, if thtr« li too 
much information. 



Cfv^k vocabulary and 
•equancat of instruction. 



c3 




Revisa araas wharalndic^^i 



Langth of instruction may 
hayato bashortanad. Numbar 
of summarias of saif *tasts may 
hava to ba incraasad. SUppla- 
mantary matarials may hava to 
baaddad. 



^38 
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Identify job(s) for which 
student will be prepared. 



. a1-4 
"1 



• 






* 


On line 














Defjiribe initruction. 












a1 



Guide for the use of author aldt in praparing CAI mat«riali. 



Discuss rationale fot " ' 


instruction. 






aM 



Identify th© need for 


instfiuction. 






\ 

a1-2 ' 



Identify the target of 


instruction. 






a1-3 
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V 



2.« 



Idantlfy dagrM td which 
initruotion trains itudcnt for^b. 



•1-6 



Prepare overview of initruction. 



a2 



ERIC 



Prepare outline of each lesson. 



a2 1 



List lessons in their 
proposed sequence. 



a2-3 



Write plan of instruction. 



a3 



Bf ief ly describe cc 
of aech lesson. 


»ntents 
a2-2 

— ^ 







Indicate LO's for each sequence 
or block of instruction. 



a3 1 



T 



*Non-HumRRO author aid. 



See :Handbook for Designers of Instructional Systems, AFP 50-58, 
July 1973. 



9 
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2.( 

I 

Otttrmlnt duration of training 
for Moh lnitruotk>nal taction. 



a3^2 



Deicrlba Instructor rtquire- 
nrMhtt/dutioi in aach instructional 
taction. ^ 

a3-3 



Datcribit madia raaourca 
raqulramants ralatad to training 
aids and facilitias. 

a3-4 



TargBt population chscription. 



. a4 



3, 



List tima conitraintf for aach taquanca, praotloa axarclia, ate. 



List the available facllltiet, the time and duration of their ute and ^ 
the luperviiory pertonnel needed. LItt all available training aidr 
for each block or unit. Lilt a\\ mediated and lupplementary 
instructional materials, their location, etc. 



ERIC 



Identify trainees' academic or 
educational level, reading level, 
verbal ability, etc. ^ 

a4-1 



Identify previous training or ^ 
related knowledge and experience 
of trainees. 

a4-2 



Identify required physical and- 
personal characteristics of trainees. 



a4-3 



Refer to entry tests. 



List courses taken and /or hours of training in specific 
prerequisite areas. 



List required physical skills and characteristics (e.g., coordination/ 
motor^ills) and personal qualities (e.g., leadership, motivation, etd.). 
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2.e 

. I 

Identify admlnlitratlve re%traint<. 



•44 



On line 

and 
Offline 



T09ting Inforrrmt 












^ttri4 in biook If 

, . • . r 

K ■ . ' 

t 






Furnish answers to test items. 

■■ . \ 

a52 


- - ■ 





Provide directions for 


administratfon. 






# 








a5-3 



Provide scoring procedures to be 


used. 






a5-4 



Administration directions. 



a6 



Lilt all tequance ranks and grades of students* 



.ERIC 
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DMcrib* toheduling procifcturM. 



Ditcuu monitoring procett. 



a62 



Provide instructions for 
maintaining attendance records. 



a63 



List tqhttdMle for using facliltltt, materlalt, lind pertonnel 
reaourcM. 



Describe monitoring requirements for 
aelf*tests and practical exercises. 



a6 4 



Provide recomm^dations for 
handling individual trainee 
differences, 

a6 5 



Describe procedures for keeping 
the student productively involved in 
the learning process. 

. a7 



Indicate recommendations for 
S^ovidlngan environment conducive 
to learning, 

a8 



^^scuss needs of exceptionally fast/slow trainees. 



For example, ways to elicit student activity. 



For example, discuss value of displaying high motivation 
of teachers and easy access to aids. 
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ii3 



• 2A 

« 

Provld«t«tchino typft, mtthodi, 
•nd t«ohni4uM. 



For exampU. IndlCAtt wayt to change pace In Initruotlon. 





> 


b ■* 




r- 




Lift prerequisites |[or course 
in terms »f TLO'f. 


V 


b1 




I- 


i 


Explaih framework of course. 


Describe time frame, prerequisite assumptions, pre-teits, if any, 
post-test requirements/criteria, and course materials. 


* b2 . 





Describe structure of course and its 
environment. 



b3 



Explain sequenige of cx)ur9e lessons, Use of aids, facilities, and 
personnel in control of course, remedial branching; if present. 



Specify personnel to contact 
when instructions pr course 
materials are not understood. 
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3.0 Outputt 



V 



Ptoducti. 








3.1 



Furniih all Instructional materials, 
•xarclsas, self-tatts, aids, etc. 



Fufnish Instructions for using 
all materials developed. 



Other documentation. 



3.2 



Prepare an outline statement of 
instructional development plan. 



Write a summary Statement 
of any deviations from plan and 
reasons for deviations. 

b 



Prepare a report detailing 
development time, costs, and 
problems/solutions. 



Describe management of course and students. 



ERIC 
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Appendlc C 

DETAILED FLOWCHARTS OF AUTHOR AJDS FOR 
.. IPISD BLOCK II. 2 ('DEVELOP TESTS) AND BLOCK 111.4 (DEVELOP INSTRUCTION) 

•*• 

The two sections of flowcharts in this appei>dix serve- three, purposes. 
First, they further clarify IPISD Blocks II. 2 and III. A. The number at the 
beginning of each flowchart refers to the appropriate place they are to be 
Inserted in the IPISI> blocks flowcharted in Appendix B. Additional block 
identification was not made because of the complexity of the flowcharts. 

The second purpose of^ the flowcharts is to provide directions for the 
Vse of the on-line authoring. aids for teat and lesson development. These 
flowcharts are detailed endugh to lead an author from thfB beginning of 
development (the learning objective) to the finished product (on-line 
for student use) . ' 

The third purpose of the flowcharts li to show the logical flow of the 
author aids (including where they fit ii^tp the larger IPISD process), so 
that conversion of the aid is possible ^^^n other computer-administered 
instructional systpiife. On-line demonstrations of the flowcharts are possible 
by accessing HumRRO lessons Inquiry 1 (test development) and Inquiry 2 (lessson 
development on the PLATO system. 

The f lowcha,rt8 in this appendix ar'e on the. pages listed below. 

Preparing CAI. Test Materials — ■ Page C-2^ , 
Preparing CAI Lesson Materials, — Page C-30 



li7 



C-1 



Il.3.2;i.1.b 



PREPARINQ CAI TEST MATERIALS 




N 



N 



Transfer block b 
from inquiry 3 and 
make the neces^ry 
changes. Refe^to 
the manual 



Obtain from the 
site director 



See site director to 
ol^tein detetet for 
tem]|x>rery storage ^ 
of test 



e 


/ Has X 
/ PLATO 6 \ 

part lesson 
\. been j 
designated 
\ 7 / 


V N 


Sea site director for 
PLATO lesson to 
be used for final 
storage of test 












i 


^ J 


* 








Y 

















ERIC 



lis 



C-2 



Typ« in whtlhtr 
or not ttudtnt It 
giv«n th# optipn 
of branching to 
tht quiz 



Study general tett 
guidelinet 



Study learning 
objective guidelines 



Enfer learning 
objective No. 1 



r 




Use learning^ 
objective prepare^ 
tion worksheet ' 




Make 
revisions 



'4 ^♦'S 





Type in number 
of answer 
alternatives 
including correct 
answer. 





Revise number 




N 


of answer 




alternatives. 






• 




1 


— . 1 



ERIC 



X Chose 

position y 




Type in answer 
alternatives including ' 
correct answer in 
posUion desired. 


• 


Specify the 
correct answer. 




/ of correct answer 

alternative / 

X ' / 












N. 












Type in answer 
alternatives including 
correct answer 
\ (position randomly 
selected) 






* 


T ^ 

./ 
















- ■ 4 — 








Y 


Type in time 
permitted;^^^ 







N 



Complete scoring 
form for item. 






Y . 


Review next 


learhing objective to 


determine action 


required. 
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• 


t 


N 


N 




<• 








i» 



Type in whether 
answer is true 
or false 




Complete scoring 
form for item. 




Review nex|^ 
learning objective to 
determine action, 
required. 




Type in time 
permitted. 




1 
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/ 




Havt 

duidellmt 
^for tntaring •niwen 
been studied 

? 



N 



Determine scoring 
options. 




Study guldelinei. 



Enter as many as 
6 correct, partially 


correct or incorrect 


answers. 




^- 








4 






? 



Type changes if 
necessary. 



r 



c-is 



159 




N 




Y, 



Type change if 
necftturVi. 



Type chanae if 
necisssary . 



(J. 
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< 



160 




ERIC 
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D 



Stop tomporarilv* 




Tvpe in lesson name 
for final storage of 
test.- 




Sign off 
K (Hit STOP 1) 



See manual for 

conversion 

instructions. 



Review test. 



See manuka! fof 
instructions. 



C^19 



Sign into 
inquiry 2 
when ready to 
continue. 



i65 



5 I CrMte during tett initructlont 




Write during test 
instructions 






Make 
revisions 



» ' 167 

C-21 



. 1 






Type in new 
. lesson name for 
/ final storage 
of test 



' C-22 



\ 



Revl«w toorini 
w6rkfh0ett 



Type In secret 
and auign score 
to each answer 



Specify minimum 
passing score 




> 



ERIC 



Will 

student review"^ 
items missed 
if test is 
passed 
? 



Select student 
review option 



C-23 



iS9 



10 



Specify review 
option desired 




i 70 



C-24 




.1* 




_Tvpe in 
option 5 




172 
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\ 



1^4 



/ 



ERIC 
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© 



See manual on 
conversion ' _^ 
instructions 



Review test 



End CAI 
test creation 





PREPARING CAI LESSON MATERIALS 



III. 4.2.3.13 




— ^ I 

Typ# in data sat name 
and latson name. 



Has reading 
difficulty level been 
decided 



N 



Determine reading level 
of trainees. 



Subtract 1 from read^l^g 
level determined. 



Study guideline describing 
modules and lessons. 




Study guidelines for 
preparing practice framei. 





Create outline for 
instruction. , 




Set "margirr; • . 
(Use the arrow keys) 



V 



Practice frame creation 




f 



Muitiple%hbice format 




Use MultipleK:hoice 
practice item - 
. worksheet. 





Use option. ' y 




Y 






m 


; ; -A 


f9 



f 



Type in number of 
answer alternatives 
including correct 
answer." 





Type in answer 
alternatives 
including correct 
answer in position 
desired. 



Type in answer . 
alternatives 
including correct 
answer in position 
randomly selected. 




Type in number . 
attempts permitted 
(13). 



n 



Revise number of 
answer alternatives. 



Specify the correct 
answer. 



Make revisions. 



J-S3 
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i 




Make revisions. 



ERIC 
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Type In foodback 
tor attempt n. 






Made revisions. 




Revise 






teedlMick jT 


y 






^ 




1 N 





er!c\ , 



\ 



C-39 i 



> 




TyP« whether 
answer is true 
or false. 






Study example. 




Y 


r ' 



















* Enter' as mdny ^ 
4 correct answers. 



f ■ 

y 




Use)cohstructe«j 
r^siSonse practice 
item 'worksheets. 



is 8 



ERIC 

!-■' N 



0-42 



Determine scoring 
options. 




Type change if. 
npceJsary. ^ 



Type change if 
necessary. , 



Type change if 
necessary. 



ERLC 




Enter as many as 
four anticipated 
wrong answers. 




, ( K3,3 




Type try again 
feedback tor 
attempt 1 



Type try again 
feedback for next 
attempt. 










Y 


Make revisions. 



















• ' 1 


t 


Y 


^ . ^ 

Make revisions. 

















V. 



194 
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/ 



\ 



K3.4 



Another ^ x j^j 
Constructed responsOy 
Ftem 
? 



K3 



i 





ERIC 
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< 






Stop tempor^rllY. 



See manual for 
-conversion 
instructions. 



Review lesson. 



See manual for 
instructions. , 



Type in new lesson 
name for fJn«rt 
storage of 
instruction, ^ 



Sign-6ff. 


— r ~ 


(Hit STOP 1) 




1 


t 




i 



^ 1 



Sign into inquiry 2 
when rbady^o 
continue. 



I. 
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1 



See manual for 

conversion 

instructions. 



Review lesson. 



End CAI lessoft 
creation. 



Stop, lesson is (inished. 





Type in new 
lesson riQHfie for 
final storage of 
instructions. 



See Appendix B 
page B-79. 
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